The first and most significant change we are making is to how participant systems provide output. In particular, participant systems are now to provide all categories that they consider to be relevant for a tweet, rather than only one. This is a direct result of our experiences from the first year, where it became obvious that some tweets just belong to multiple categories. We will also provide updated metrics to account for this change – in effect, the more overlap between the assessor-selected categories and the system-selected categories the better.

From a submission perspective, this means that participants now return a comma-delimited list of information types for each tweet rather than the most likely information type.

Secondly, we will be making some minor changes to the underlying information ontology based on our experiences and feedback received. Here are the changes being proposed

Entirely Removed Categories:

• Other-Unknown: This is a very rare category that was added so assessors had somewhere to place tweets that they just could not decide upon. Tweets that ended up here were either incomplete or needed some additional context to understand. This is not something that we want participants to consider, so we will be removing it from the ontology.

Categories with Altered/Refined Meaning:

• Other-PastNews -> Other-ContextualInformation: This is a refinement of the old PastNews category to reflect how it was used. In practice, the PastNews category represented stories that mentioned past events that happened prior to the current event. For example, for an earthquake, other past earthquakes in the same region. We are changing the name to make this interpretation clearer. This category now encompasses all tweets that provide contextual information that makes it easier to understand the event.

• Report-ContinuingNews -> Report-News: During judging, the Report-ContinuingNews was used basically as a catch-all for any current news report. This change is simply a clarification of that.
• Other-KnownAlready -> Report-OriginalEvent: For some events, we provided information in the event description that the user (emergency management officer) already knew. This was nearly always some variant of ‘the event occurred’. For example, a report of an earthquake is typically not useful, because the response officers will already know because they have physical sensors that report the event faster. The 2018 category Other-KnownAlready was meant to capture tweets that contained this known information, however it was confusing to some participants. Hence, we care altering it to Report-OriginalEvent, which simply represents that the tweet reports the occurrence of the initial event.
• Report-SignificantEventChange -> Report-NewSubEvent: One of the main confusions that turned up when judging was the distinction between Report-SignificantEventChange and Report-EmergingThreats. To make the distinction clearer (particularly now participants can select multiple categories), we are re-naming Report-SignificantEventChange to Report-NewSubEvent. Report-NewSubEvent represents any tweet that reports the instance of a new occurrence that is so significant that it needs to be considered separately. For example, for an earthquake this might be aftershocks, or for a terrorist attack, this could be follow-up attacks.