Contest

Dataset

Evaluation protocol

The fire detection accuracy of the competing methods will be evaluated in terms of Precision and Recall. To formalize these metrics, it is necessary to define the sets of true positives ($TP$), false positives ($FP$) and false negatives ($FN$). Our test set contains both positive and negative samples, i.e. video depicting fire events or not, respectively. Each positive video shows one fire event only. Each prediction is evaluated by comparing the fire start instant $g$ with the detection one $p$. We indicate as the fire start instant the first frame in which it is visible. It is manually labelled by the human operator after visioning the whole video, thus it is a ground truth. Notwithstanding this, we consider that automatic methods could anticipate human detection by a few seconds.

Due to this, we define:

• $TP$: all the detections occurring in positive videos at $p \ge max(0,g- \Delta t)$;
• $FP$: all the detections occurring at any time in negative videos or in positive videos at $p < max(0,g- \Delta t)$;
• $FN$: the set of positive videos for which no fire detection occurs.

with $\Delta t$ equal to 5 seconds.

Defined these sets for $TP$, $FP$ and $FN$, we can compute the Precision ($P$) and Recall ($R$) with respect to the number of true positives ($|TP|$), false positives ($|FP|$) and false negatives ($|FN|$):

$ P= \frac{|TP|}{|TP|+|FP|} $

$ R= \frac{|TP|}{|TP|+|FN|} $

Precision assumes values in the range [0,1] and measures the capability of the methods to reject false positives; the higher is $P$, the higher the specificity of the method. Recall assumes values in the range [0,1] and evaluates the sensitivity of the method to detect fire; the higher is $R$, the higher the sensitivity of the method.

The fire notification time of a method for the i-th true positive is $p_i$ , while the fire start time is $g_i$. Therefore, the notification delay $d_i$ on the i-th sample is:

$ d_i= |p_i - g_i| $

The average notification delay $D$ is defined as the ratio between the sum of all the notification delays $d_i$ and the total number of true positives $|TP|$:

$D=\frac{\sum_{i=1}^{|TP|}{d_i}}{|TP|}$

The smaller is $D$, the faster the notification of a fire. To obtain a value in the range [0,1], we can compute the normalized average notification delay $D_n$ as follows:

$D_n=\frac{max(0;60-D)}{60}$

Then, we compute the processing frame rate ($PFR$), namely the average number of frames processed by the method in one second (on a target GPU that we will adopt for our experiments). In particular, being $N$ the total number of frames processed in the test set and $t_i$ the processing time in seconds for a single frame, the $PFR$ is computed as follows:

$PFR=\frac{1}{\frac{\sum_{i=1}^{N}{t_i}}{N}}$

The higher is $PFR$, higher is the processing speed of the method. To normalize this value with respect to the minimum processing frame rate needed to achieve real-time performance, namely $PFR_{target}$, we compute the $PFR_{delta}$ score as follows:

$PFR_{delta}=max(0;\frac{PFR_{target}}{PFR}-1)$

Finally, we measure the memory usage ($MEM$), namely the memory in GB occupied by the method on the target GPU. The lower is $MEM$, lower is the necessary memory on the processing device. To normalize this value with respect to the maximum GPU memory available on the target processing device, namely $MEM_{target}$, we compute the $MEM_{delta}$ score as follows:

$MEM_{delta}=max(0;\frac{MEM}{MEM_{target}}-1)$

We define the ranking of the contest according to Fire Detection Score ($FDS$), computed as follows:

$FDS=\frac{P \cdot R \cdot D_n}{(1+PFR_{delta}) \cdot (1+MEM_{delta})}$

The method which achieves the highest $FDS$ will be the winner of the ONFIRE 2023 Contest, since it will demonstrate the best trade-off between fire detection accuracy, notification promptness and required processing resources.

NOTE: We will consider a processing device with 128 CUDA cores and 4 GB of memory, so the values for $PFR_{target}$ will be 10 and $MEM_{target}$ will be 4.

Rules

1. The deadline for the submission of the methods is 21st July, 2023. The submission must be done with an email in which the participants share (directly or with external links) the trained model, the code and the report. Please follow the detailed instructions reported here.
2. The participants can receive the training set and its annotations by sending an email, in which they also communicate the name of the team.
3. The participants can use these training samples and annotations, but also additional videos.
4. The participants must submit their trained model and their code by carefully following the detailed instructions reported here.
5. The participants are strongly encouraged to submit a contest paper to ICIAP 2023, whose deadline is 28th July, 2023. The paper can be submitted through Easychair. Authors can find complete instructions of how to format their papers here. The maximum number of pages is 12 including references (minimum number is 6). Accepted papers will be included in the ICIAP 2023 Workshops Proceedings, which will be published by Springer in the Lecture Notes in Computer Science series (LNCS). The contest paper must be also sent by email to the organizers of this contest. Otherwise, the participants must produce a brief PDF report of the proposed method, by following a template that can be downloaded here.

Instructions

The methods proposed by the participants will be executed on a private test set. To leave the participants totally free to use all the software libraries they prefer and to correctly reproduce their processing pipeline, the evaluation will be done on Google Colab (follow this tutorial) by running the code submitted by the participants on the samples of our private test set.

Therefore, the participants must submit an archive (download an example) including the following elements:

• A Python script test.py, which takes as input the folder of the test videos (--videos) and produces as output a TXT file with the detection time (or nothing) for each video in a folder (--results). Thus, the script may be executed with the following command:
  python test.py --videos foo_videos/ --results foo_results/
• A Google Colab Notebook test.ipynb, which includes the commands for installing all the software requirements and executes the script test.py.
• All the files necessary for running the test, namely the trained model, additional scripts and so on.

The provided sample test.py also includes the reading of the TXT files with the annotations. Each file includes the fire ignition instant or nothing if it is a negative video. The files of the results will be formatted exactly in the same way. The provided sample test.py includes the writing of the files with the results.

The submission must be done by email. The archive file can be attached to the e-mail or shared with external links. We strongly recommend to follow the example of code to prepare the submission.

The participants are strongly encouraged to submit a contest paper to ICIAP 2023, whose deadline is 28th July, 2023. The paper can be submitted through Easychair. Authors can find complete instructions of how to format their papers here. The maximum number of pages is 12 including references (minimum number is 6). Accepted papers will be included in the ICIAP 2023 Workshops Proceedings, which will be published by Springer in the Lecture Notes in Computer Science series (LNCS). The contest paper must be also sent by email to the organizers of this contest.

Results

Before the deadline, 17 teams sent an official request for participating to the contest from all over the world: 9 from Europe, 5 from Asia, 3 from North America. Finally, 8 teams submitted a valid method to the contest, reported in alphabetic order: AI-KP, FIREBUSTERS, INTELLIGENT SYSTEMS LAB, IPLAB, KIOS-EVAI, MADFIRE, SLOWFASTONFIRE, STATE KEY LABORATORY OF FIRE SCIENCE. The test set has been balanced with the same number of positive and negative videos, in order to guarantee the same conribution of $P$ and $R$ on $FDS$.
In the following table, the final ranking of the contest is reported:

$Rank$	$Team$	$P$	$R$	$D_n$	$FDS$
1	FIREBUSTERS	0.8115	0.9802	0.9185	0.7306
2	SLOW FAST ONFIRE	0.7748	0.8736	0.8314	0.5492
3	MADFIRE	0.5130	0.8876	0.9403	0.4282
4	INTELLIGENT SYSTEMS LAB	0.2864	0.9672	0.9755	0.2702
5	STATE KEY LABORATORY OF FIRE SCIENCE	0.3460	0.9605	0.7701	0.2559
6	IPLAB@UNICA	0.3148	0.5312	0.9598	0.1605
7	AI-KP	0.4595	0.4250	0.8020	0.1566
8	KIOS-EVAI	0.4255	0.1980	0.5558	0.0468