To fix human attack on the Hugo awards, you need humans

Topic: 

I wrote earlier on the drama that ensued when a group of SF writers led a campaign to warp the nomination process by getting a small but sufficiently large group of supporters to collude on nominating a slate of candidates. The way the process works, with the nomination being a sampling process where a thousand nominators choose from thousands of works, it takes only a 100-200 people working together to completely take over the process, and in some cases, they did -- to much uproar.

In the aftermath, there was much debate about what to do about it. Changes to the rules are in the works, but due to a deliberate ratification process, they mostly can't take effect until the 2017 award.

One popular proposal, called E Pluribus Hugo appeals, at least initially, to the nerdy mathematician in many of us. Game theory tries to design voting systems that resist attack. This is such a proposal, which works to diminish the effect that slate collusion can have, so that a slate of 5 might get fewer than 5 (perhaps just 1 or 2) onto the ballot. It is complex but aimed to make it possible for people to largely nominate the same way as before. My fear is that it modestly increases the reward for "strategic" voting. With strategic voting, you are not colluding, but you deliberately leave choices you like off your ballot to improve the chances of other choices you like more.

The bigger flaw, in my mind, is that it misunderstands battle against a determined and intelligent opponent. It's actually very hard to fight a battle against human attackers with algorithms, no matter how clever. Even cryptography, the greatest example of using algorithms to defend against human attackers, knows this, and knows that the protocols must be dynamic, quickly changing in response to attack. Human against human. In the military, they call this the OODA loop, and they know that nothing else works. This is not a military battle of course, but all human systems of justice and politics today rely on human judgment as the main or final arbiter. It is hubris to imagine success will come any other way.

How to add human judgment?

The problem with human judgment is it acts capriciously. It is even corrupted. You don't know if you can trust the humans, or that even if you trust them, they will act well. So proposals to just have a court resolve things, as we do in the "real" world, are unlikely to win much support.

I suspect the answer might lie in mostly using the human judgment of the members of the World SF Society, whose award this is.

Because of conflicts with Singularity University's closing ceremony and reunion this weekend, I will not make the convention and so it is not practical for me to submit a proposal. Nonetheless, here is the best effort at what one might be:

A write-in final ballot, and some extra help.

The simplest amendment would allow people to make write-in selections on their final ballot, in any slot on the ballot. This is nothing new -- the nomination ballot is a write-in and write-in is popular around the world. It would actually be type-in as 99% of ballots are submitted online. It's a surprisingly small amount of extra work to tabulate. No matter what attackers might do to the nomination process, even completely swamping all the official final ballot candidates, it can be fixed by the voters in the final ballot by the voters -- in theory.

I say in theory, because write-ins rarely get enough support to win elections. That may be, on its own, a good thing, for it means that correction would only happen in extreme cases, such as these. To make this a sharper weapon, you may need something else.

Generally, this is the reason systems have write-in -- it's the last minute catch-all fix for any problem in the nomination system that was not expected, and which only works in the case of extreme problems.

Simple approach: just publish all below-the-bar nominees with the ballot

For those who don't like the idea of a committee even making any judgments at all, one could simply change tradition (not the rules) to have the list of candidates who did not make the nomination cut published with the ballot. Today they are published after the ceremony (as required.) The amount of information that might affect the voting (other than deliberately through write-in) in this document is very, very minor, if present at all. (You learn who withdrew a nomination, and if some candidate had a lot of just-missed works.)

In this approach, there is no committee action. All judgment is in the hands of the fans, using write-in. That's sort of the purpose of write-in.

More accurate but controversial -- having a committee study the nominations but only release the names of the missing nominees

To fully enable the power of the write-in, what fans want to know is "Which actual nominees from a fair nominating system did not make the final ballot due to the attack upon it?" If they knew that, with conviction, they could use write-in to vote for those candidates, as though the improperly nominated candidates were not there at all.

One way to do this would be to have a special Hugo award subcommittee which is created when there is some form of decision that the process was corrupted. This special group would have confidential access to all the nominating ballots. It could consider all the ballots and other evidence, and release the list of candidates who would have made the ballot absent the corruption. It could also -- though this perhaps goes further than we need -- release the names of candidates who are only on the ballot through an inappropriate process. It does need to know at least the number of these that exist in order to make the first list, but it does not need to release them.

How might this special committee come into being? It might be possible to do it with no rule change -- just a delegation by the main Worldcon committee or its Hugo subcommittee. there might also be a rule change to decide how and when to create it, and who might serve on it. For example, it could be written that a petition from 20% of the members of the convention calling for a committee (a "fannish inquisition", if you will) and naming 5 members to serve on it would create the group, though you do need to deal with competing attempts to create groups.

Either way the group could do nothing more than suggest, with some authority, some names. The rest would be entirely done by the voters. Nonetheless, there is a large distrust of any centralized human judgement in the Hugo system, so this is less likely to get approved.

Special advantages of this proposal

It may be the case that no constitutional rule change is necessary to have the review of the balloting and publication of candidates be done. The language demanding confidentiality from the public of nomination information is ambiguous. There is only a clause demanding it be published within 90 days of the convention, but it does not bar other publication, though tradition has the information not released until right after the awards. (If it's argued this is a hard limit the ballot would have to go out 90 days before the ceremony.)

As such, the examination of the ballots and publication of the names could have been done in 2015, or could be done in any future year, if the rules don't speak to that.

Here's where it gets tricky. The changing of the rules (such as to allow write-in) requires ratification a year later. It is possible, though a modest hack of the spirit of the rules, to imagine that if write-in were added in one year (such as 2015) that it could be used in 2016. That's because it is not necessary to do the final tally and present the awards at the convention, and the new rule for tallying could be ratified at the start of the convention. After the convention was officially closed, the Hugo tallying and ceremony could be done under the new rules. (Of course, you would tally under both systems in advance, and even engrave 2 plates when the two systems gave different results, then quickly re-run the proper tally and affix the right plates. In reality there would be no plates to engrave without ratification as it would be all No Award.)

This is, of course, a hack of the rules. But that's part of the philosophy that you fight human attackers with human response. Putting out a final ballot that has write-in spots as well as official nominees, along with "theoretical instructions for use if write-in is ratified" is not in the spirit of the system, which was designed to be slow to change. But the WSFS constitution, is not, as the old saying goes, a suicide pact. Nonetheless it keeps to the goal of the amendment system that any change must pass muster at two worldcons, not just one.

Human defence is a great deterrent

It is important to note that when you fight human attack with human defence, you gain a powerful deterrent. If the attacker knows you can counter what they do, it becomes far less worthwhile for them to do the attack in the first place. Here, slate collusion could put some names on the final ballot, but little else. As such, it's a lot less worth organizing and certainly less worth getting a lot of people to buy supporting memberships to execute, if that is what is needed to be done. Why spend money when you know you will lose?

Actual amendment language

The main amendment language would simply add a section 3.10.6 to say:

Voters may also write in their personal choice of candidate, so long as said candidate is eligible for nomination in that category and was published on a list of all nominees published by the convention, and so long as the total number of ranked candidates or written-in choices in that category does not exceed the number its official candidates on the final ballot, with "no award" not considered part of either count.

Further:

The convention committee shall publish, along with the ballot, a list of all nomination results, including the total number of nominations for all works which did not qualify for the final ballot but which received more than 10 nominations.

That's it -- simple, but returning power to the fans. It is too late, though, barring amending another proposal to be just this.

Comments

I'm afraid I see several flaws in this solution, beginning with the creation of a subcommittee "when there is some form of decision that the process was corrupted." The entire argument from the side attempting to do the corruption is that a secret elitist cabal is controlling the Hugo process and preventing their obviously deserving works from getting on the ballot. If you're willing to publicly proclaim that the final ballot is corrupted due to slate voting and appoint a subcommittee to publish the names of candidates who would have made the ballot had it not been corrupted and thus confirm their paranoia, then you might as well take the much easier solution of merely throwing out the corrupted ballots (which you would have identified through the same process), refunding the memberships of those who submitted such ballots, and publishing a ballot that was not corrupted. (This has the added advantage of not requiring the Business Meeting to act, as the convention committee could choose to do so on its own.)

Even if the process were to implement your suggested added overhead of establishing a subcommittee, doing the math without the corrupted ballots, and publishing the names of the candidates that would have been on the final ballot had it not been for the slate voting, this solution goes on to assume that those using the write-in option would vote solely for those works that the subcommittee identified and not, say, for the works they originally nominated that would never have made the ballot. Since you can't control for this phenomenon, the concentrated slate voting is still likely to prevail.

Finally, your 2016 scenario does not work because it would require the next convention to put the mechanism in place of establishing a subcommittee to identify the number of corrupt ballots and announce the alternative candidates. (Merely listing the next five candidates on the ballot wouldn't work because not all categories would necessarily be affected by the slate voting.) This would have the same problem as previously stated about people not concentrating their write-in votes. I also find it hard to believe that you have ever been to a Worldcon Business Meeting if you think that the ratification of such a controversial issue would conceivably take place on the first day of the convention, and in enough time for people to redo all of the math of the final ballots, or that the convention would be willing to take a chance on it passing and have substitute plaques available and in enough time to have the plaques engraved and affixed locally. Moreover, those who might conceivably win under the new math would not know they were even nominees, let alone winners, and would not necessarily even be present at the ceremony let alone have all of the kudos of being a nominee and having entree into nominee-related events.

Hate to shoot this down but I don't think it takes into account all of the real-world issues.

Throwing out the ballots is a big thing -- and would require a rules change to allow it (or at least many would argue that it would, especially those whose ballots were thrown out.)

I am pretty sure what you describe would not happen if this had been done this year. The committee examining the ballots would have re-run them without the slate ballots (defined as any ballot which voted for more than 3 slate candidates in the category in question, for example.) They would have published the list of those who now make the final ballot. Everybody would know what the slate candidates were and which ones got on the final ballot without a slate, and they could do write-ins accordingly.

For example, we would see the current best novel ballot and presumably two new additions published by the committee -- because everybody knows due to wide publicity that Skin Game and Dark Between the Stars. Voters who wished to consider the ballot as consisting of Ancillary Sword, Goblin Emperor and the 3 body problem and the two new listed novels, and would be free to enter a ballot made of nothing but those 5. (They would not list the 2 slate candidates, even after no award, which is the right approach. It is as though they were not there.)

My 2016 scenario should work as follows:

  • The amendment is passed but not ratified. The committee judges it is likely to be ratified (one presumes.)
  • The committee publishes a Hugo ballot with an ability to write in selections. It explains that if the amendment is ratified, the write-ins will be used. If the amendment is not ratified, it explains it will use rankings only on official candidates.
  • It might optionally let you rank both your write-ins and all official candidates if this is your choice, and simply disregard either the write-ins if the amendment is not ratified, or those ranked below 5th if the amendment is ratified.
  • After the ratification vote, it would run the calculation and engrave the plaques for the ceremony. Or it may just have engraved two sets of plaques if that's really an issue.
  • In 2017 and later, the modestly confusing emergency dual instructions would be replaced with only one course of action.

I think in reality we would have a very good idea if ratification were likely. Those named by the special committee as missing nominees would decide whether to travel to the convention or not -- their choice, of course -- but this does not affect the awards. All the calculations can be done long in advance, and if you really think engraving will take so long you have many options, including adding the plaque later, or making two plaques when there is a difference (there will not always be a difference) or doing cheap-ass engraving, which we all know can be done in 10 minutes at any engraving shop, and replacing it later.

The logistics are actually pretty easy. The instructions need not be that confusing, "Put a number rank next to all candidates you wish to rank, including up to 5 write-ins, and we will calculate the winner based on the rules ratified at the convention. Thanks to Australian ballot you can do your ranking without having to predict if the new rules will be ratified."

No, the only questions in my mind are:

  • Does having a Hugo subcommittee reveal the pushed-out nominees violate the WSFS constitution, or will a significant set of people strongly feel it does?
  • How, for 2 years down the road, would you want to formalize any change that allows the fair creation of such a committee.

Remember, the committee's power is extremely small -- it simply is able to disclose a few details from the nomination summary a bit early, if it finds corruption of the process. The information it discloses is required to be published after the awards anyway. With such limited ability, it should not be hard to figure out how to empanel it fairly, in the rare cases it might need to be done.

That for those who are bothered by having a group figure out the "real" ballot and the contention that might come out of it, a simpler proposal is just to publish the whole below-the-cutoff nomination list in advance, rather than after the ceremony. (Totals for the works that made the ballot would wait until after the ceremony.)

There is an argument the constitution would forbid that more than 90 days from the ceremony, but if so, it's just a later balloting and this could be fixed with an amendment.

Add new comment