Internationally, civil protection, police forces and emergency response agencies are under increasing pressure to more quickly and effectively respond to emergency situations. Moreover, such emergencies are common and recurring. For example, 50,000 people per-year on average die during natural disasters internationally.

The mass adoption of mobile internet-enabled devices paired with wide-spread use of social media platforms for communication and coordination has created ways for the public on-the-ground to contact response services. Moreover, a recent study reported that 63% of people expect responders to answer calls for help on social media.

With the rise of social media, emergency service operators are now expected to monitor those channels and answer questions from the public However, they do not have adequate tools or manpower to effectively monitor social media, due to the large volume of information posted on these platforms and the need to categorise, cross-reference and verify that information.

The Text Retrieval Conference is a combined conference and evaluation campaign that aims to encourage research into information retrieval technologies from large test collections. It is co-sponsored by the National Institute of Standards and Technology (NIST) Information Technology Laboratory's (ITL) Retrieval Group of the Information Access Division (IAD), and has run annually for over 25 years.

TREC consists of a set tracks, areas of focus in which particular retrieval tasks are defined. The tracks serve several purposes. First, tracks act as incubators for new research areas: the first running of a track often defines what the problem really is, and a track creates the necessary infrastructure (test collections, evaluation methodology, etc.) to support research on its task. The tracks also demonstrate the robustness of core retrieval technology in that the same techniques are frequently appropriate for a variety of tasks. Finally, they make TREC attractive to a broader community by providing tasks that match the research interests of more groups.

Incident Streams is a TREC track designed to bring together academia and industry to research technologies to automaticaly process social media streams during emergency situations with the aim of categorizing information and aid requests made on social media for emergency service operators.

The TREC-IS task is to produce a series of curated feeds containing social media posts, where each feed corresponds to a particular type of information request, aid request, or report containing a particular type of information. These "types" are defined based on existing hierarchical incident management information ontologies, such as MOAC (Management of a Crisis), For instance, for a flash flooding event, feeds might include, "requests for food/water", "reports of road blockages", and "evacuation requests". In this way, during an emergency, individual emergency management operators and other stakeholders can register to access to the subset of feeds within their domain of responsibility providing access to relevant social media content.

TREC-IS is divided into editions. For each edition, the organisers release a dataset of events for testing systems. Participants submit the output of their system(s), which are then evaluated by the organisers. TREC-IS has had six prior editions, 2018-B, 2019-A, 2019-B, 2020-A, 2020-B and 2021-A. Associated publications for each edition can be found below. The organiser overview paper is a good place to start, as it summarizes the task, datasets and system performances. The task guidelines are what was provided to the participants in terms of instructions. The notebook papers summarize the approaches taken by different research groups to the task. 2021-B was the last edition of TREC-IS and has concluded. However, you can still participate in 2021-B via our online leaderboard.

Task Guidelines

Task Guidelines 2021

Interested in Participating?

2021 Instructions Dataset and Labels TREC-IS Online Leaderboard

Overview

2020 Overview (ISCRAM Version)

Task Guidelines

Task Guidelines 2019-B Task Guidelines 2019-A

Participant Notebooks

Beijing University of Technology New Jersey Institute of Technology University of Glasgow

Overview

2019 Overview (ISCRAM Version)

Task Guidelines

Task Guidelines 2019-B Task Guidelines 2019-A

Participant Notebooks

RCAIT Chonbuk National University Carnegie Mellon University German Aerospace Center (DLR) University of Chinese Academy of Sciences Indian Institute of Technology (BHU) Universite de Toulouse UPS-IRIT University of Alicante University College Dublin

Overview

2018 Overview (ISCRAM Version)

Task Guidelines

Task Guidelines 2019-B

Participant Notebooks

Beijing University of Technology Chonbuk National University German Aerospace Center (DLR) Indian Institute of Technology (BHU) Toyohashi University of Technology NHK Science and Technology Universidad de Jaen Paderborn University