This page provides instructions for how to participate in the 2020-B edition of TREC-IS. In particular, we cover how to participate, where to get the data upon which your system will be evaluated, and the timeline for submission.

Your one-stop-shop for information on the task and how to particpate is the guidelines document for 2020-B (Updated 26/08/2020):

You may also want to read up on past editions to see what previous participants have tried:

Submission instructions and the link to the submission form will be made available in September.

There are three tasks as described below in 2020-B, which are the same as 2020-A. For all tasks, we evaluate systems based on their categorization and prioitization performance over different event streams.

Task 1 and 2 : We use 14 events crawled during 2020. The topic description file for each these 14 events are provided below:

Task 3 : The overall 'event' is the COVID-19 outbreak. This is split into nine streams, each from a different location. Topics:

Task 1 and 2 :For each of the aforementioned 14 events we provide a stream of Twitter tweets. You should assign categories for all tweets in each stream. The tweet stream can be downloaded in the same way as the past events using the downloader tool. In this case, the request key should be 'trecis2020-B' (no quotes).

Task 3 : For nine locations we provide a stream of Twitter tweets. You should assign one of more of the 9 information type labels and a priority label. The request key for the nine events is 'trecis2020-B-covid' (no quotes). The number of tweets that should be downloaded is 329,717 (50k per event, downloading may take some time, mail me if you have issues).

We are running three tasks as part of 2020-B that you can participate in. Tasks 1 is the same task as in previous editions (information categorisation, high-level). Task 2 is the same as task 1 but with a reduced category set. It is designed for participants who only want to focus on categories that are likely to contain actionable information. Note that if you participate in Task 1, we will also report your performance on Task 2. Task 3 is the COVID-19 task. It uses a different event to Tasks 1 and 2 and uses a further reduced category set.

In this task you will process a stream of tweets from different events and you need to assign one or more of 25 information type labels and one priority label (Critical, High, Medium or Low) for each tweet. This task uses 14 crisis events. The information type categories are:

• Request-GoodsServices
• Request-SearchAndRescue
• Request-InformationWanted
• CallToAction-Volunteer
• CallToAction-Donations
• CallToAction-MovePeople
• Report-FirstPartyObservation
• Report-ThirdPartyObservation
• Report-Weather
• Report-Location
• Report-EmergingThreats
• Report-NewSubEvent
• Report-MultimediaShare
• Report-ServiceAvailable
• Report-Factoid
• Report-Official
• Report-News
• Report-CleanUp
• Report-Hashtags
• Report-OriginalEvent
• Other-ContextualInformation
• Other-Advice
• Other-Sentiment
• Other-Discussion
• Other-Irrelevant

In this task you will process a stream of tweets from different events and you need to assign one or more of a reduced set of 12 information type labels and one priority label (Critical, High, Medium or Low) for each tweet. This task uses 14 crisis events. The information type categories are:

• Request-GoodsServices
• Request-SearchAndRescue
• Request-InformationWanted
• CallToAction-Volunteer
• CallToAction-MovePeople
• Report-FirstPartyObservation
• Report-Location
• Report-EmergingThreats
• Report-NewSubEvent
• Report-MultimediaShare
• Report-ServiceAvailable
• Other-Any

In this task you will process a stream of tweets about the COVID-19 outbreak in different affected regions and you need to assign one or more of the full 25 information type labels (the same as for Task 1) and one priority label (Critical, High, Medium or Low) for each tweet. The information type categories are:

• Request-GoodsServices
• Request-SearchAndRescue
• Request-InformationWanted
• CallToAction-Volunteer
• CallToAction-Donations
• CallToAction-MovePeople
• Report-FirstPartyObservation
• Report-ThirdPartyObservation
• Report-Weather
• Report-Location
• Report-EmergingThreats
• Report-NewSubEvent
• Report-MultimediaShare
• Report-ServiceAvailable
• Report-Factoid
• Report-Official
• Report-News
• Report-CleanUp
• Report-Hashtags
• Report-OriginalEvent
• Other-ContextualInformation
• Other-Advice
• Other-Sentiment
• Other-Discussion
• Other-Irrelevant

Participants can use all of the previous events and associated annotations to train their systems. These can be accessed from the download page. When downloading tweets for training, we recommend using the 'past' or 'past-covid' request key with the downloader.

To help guide systems, we also provide a summary of who the end user is and what they might be interested in seeing for each event type. These are given to our human assessors.