The new 2019 edition of the TREC-IS track makes some changes to the task and resources available based on lessons learned from the first year. On this page we summerize the main changes to help participants transition between years.
The first and most significant change we are making is to how participant systems provide output. In particular, participant systems are now to provide all categories that they consider to be relevant for a tweet, rather than only one. This is a direct result of our experiences from the first year, where it became obvious that some tweets just belong to multiple categories. We will also provide updated metrics to account for this change – in effect, the more overlap between the assessor-selected categories and the system-selected categories the better.
From a submission perspective, this means that participants now return a comma-delimited list of information types for each tweet rather than the most likely information type.
Secondly, we will be making some minor changes to the underlying information ontology based on our experiences and feedback received. Here are the changes being proposed
Entirely Removed Categories:
Categories with Altered/Refined Meaning: