Emergencies
Card image

Internationally, civil protection, police forces and emergency response agencies are under increasing pressure to more quickly and effectively respond to emergency situations. Moreover, such emergencies are common and recurring. For example, 50,000 people per-year on average die during natural disasters internationally.

Social Media
Card image

The mass adoption of mobile internet-enabled devices paired with wide-spread use of social media platforms for communication and coordination has created ways for the public on-the-ground to contact response services. Moreover, a recent study reported that 63% of people expect responders to answer calls for help on social media.

The Challenge
Card image

With the rise of social media, emergency service operators are now expected to monitor those channels and answer questions from the public However, they do not have adequate tools or manpower to effectively monitor social media, due to the large volume of information posted on these platforms and the need to categorise, cross-reference and verify that information.

Introducing TREC

Text Retrieval Conference

The Text Retrieval Conference is a combined conference and evaluation campaign that aims to encourage research into information retrieval technologies from large test collections. It is co-sponsored by the National Institute of Standards and Technology (NIST) Information Technology Laboratory's (ITL) Retrieval Group of the Information Access Division (IAD), and has run annually for over 25 years.

TREC Tracks

TREC consists of a set tracks, areas of focus in which particular retrieval tasks are defined. The tracks serve several purposes. First, tracks act as incubators for new research areas: the first running of a track often defines what the problem really is, and a track creates the necessary infrastructure (test collections, evaluation methodology, etc.) to support research on its task. The tracks also demonstrate the robustness of core retrieval technology in that the same techniques are frequently appropriate for a variety of tasks. Finally, they make TREC attractive to a broader community by providing tasks that match the research interests of more groups.

Incident Streams Track

TREC-IS

Incident Streams is a TREC track designed to bring together academia and industry to research technologies to automaticaly process social media streams during emergency situations with the aim of categorizing information and aid requests made on social media for emergency service operators.

Task Definition

The TREC-IS task is to produce a series of curated feeds containing social media posts, where each feed corresponds to a particular type of information request, aid request, or report containing a particular type of information. These "types" are defined based on existing hierarchical incident management information ontologies, such as MOAC (Management of a Crisis), For instance, for a flash flooding event, feeds might include, "requests for food/water", "reports of road blockages", and "evacuation requests". In this way, during an emergency, individual emergency management operators and other stakeholders can register to access to the subset of feeds within their domain of responsibility providing access to relevant social media content.

Card image Card image
Get Started

To get started read the detailed the Task Guidelines and have a look at the Ontology of information types

Visualisation
Card image

2018 Edition Information

2018 Summary

A single task is was run for the first year of the track (2018): classifying tweets by information type (high-level). Task guidelines are provided, along with a training dataset where information types and priority levels have been manually annotated for a set of tweets. 'User' profiles are provided for each event type, defining what is relevant for each event type (this is what the assessors are provided when judging).

Task Information
Training Data
Event Type Profiles
Test Data

Instructions for downloading the test dataset, including the topics, ontology and tweets can be found at the link below.

A summary of the evaluation metrics for the 2018 edition is summarized at the following link.

2018 Outcomes

The 2018 Edition of TREC-IS has finished. Across you can find the reports, presentations, publications, open source systems and resources produced from this edition

Reports and Publications

### Under Construction ###

Evaluation Resources
Open Source

Interested?

Track Status
Card image

Participant runs have been evaluated and the results returned to the participants. TREC-IS will run again in 2019.

Timeline
  • Guidelines released: 29/05/2018
  • Training dataset release: 30th June 2018
  • Test dataset release: 8th August 2018
  • Run assessing period: 5th - 29th September 2018
  • Scores returned: 15th October 2018
  • Notebook papers due: 31st October 2018
  • TREC: 14th-16th November 2018
Get Updates
Card image

You can join our Google Group with the button below to recieve updates and ask questions about the track.

Links
Supported By
Card image Card image Card image