For the 2020-A edition of the track, participants can submit the output of different versions of their solution to the organisers for each task. All submissions will be evaluated against human annotations of the tweets for the test events. Evaluated submissions will be ranked based on a target metric, which promotes the correct classification of tweets that contain critical information (we mostly care about getting actionable information to the response officer, and less about other categories like sentiment or general news reporting). A participating group can provide a maximum of 4 new submissions per task. This is to stop a participant from spamming hundreds of submissions with minor parameter tweaks. You may also submit the output of systems from previous TREC-IS editions, these do not count towards the 4 system limit (if you participated in previous editions then please do submit the output of those older systems, so we can better track performance across editions).

The advantage of submitting runs is that you will get performances returned to you in June and your system will appear in an overview paper that the track organisers will produce. The evaluation scripts and human labelled data will be released later (likely in July) for those people that could not participate. The organisers will also archive your submitted runs, and will distribute them on request. This helps promote reproducable research and enables furture systems to be compared with what the state-of-the-art was at submission time.

To construct a submission, you need to have a system that outputs predicted information categories and priority levels for each of the tweets in each of the test events. In effect, we expect participants to have a system that ingests and processes the tweet stream for each event in time order, and will emit one line in a text file per tweet containing the following information separated by tabs:

• The incident identifier (the contents of the "num" tags in the topic files for Task 1/2 and/or Task 3.
• A literal string "Q0" (its to support compatability with an old piece of software)
• The tweet ID of the tweet, an 18-19 digit number
• The tweet number in the stream, sometimes referred to as the rank. Start at 1 for each event and count up.
• The importance/criticality label, either Low, Medium, High or Critical. (Note that this is different from prior editions where a score rather than a label was submitted!)
• The information types within the ontology for the current tweet. The valid information types depend on which task you are submissing for. This should be a comma-delimited list as illustrated below.
• The runtag, this should be a unique identifier for your system and institution. For example, UoGlasgow-DLRun1.

To create a run submission, you need to create a ZIP file with the name '[runtag].2020A.task[X].zip' (replace [runtag] with the name of your run as defined in the last row of your run) and [X] with the task number, where the ZIP contains two files:

• Your run file (see the run format description above), where the filename is [runtag].2020A.task[X].run.
• A readme.txt file, containing the following:
• name of your company or university
• name of the person submitting the run
• the email address of that person.
• either 'academic', 'public sector', or 'industry'.
• the run's runtag
• 2-3 sentences describing the key aspects of your approach (it helps us get an overview of what people tried)

For the 2020-A edition, submissions are made to the University of Glasgow via a file transfer service:

• http://transfer.gla.ac.uk/

On this page, select 'Drop-off'. You will be asked to verify your contact details, enter your details and press 'Send Confirmation'. You will shortly recieve an automatic email entitled '[UofG Transfer] You are trying to drop off some files'. Press the link in the email, and you will be taken back to the transfer service, with a box open asking for recipients, enter the following details:

• Name: Richard McCreadie (TREC-IS)
• Email: richard.mccreadie@glasgow.ac.uk

In the 'Short note to the Recipients:' box, enter 'TREC-IS 2020-A Submission'. Then press 'Click to Add Files or Drag Them Here' and select your submission zip. If you have multiple submissions, you can add them all here. Once done, press 'Drop-off Files'. You should see a final screen indicating that your files have been successfully sent. You will also recieve an email some time later when I download your submission, so you know I got it.