Contest

Dataset

Evaluation protocol

The fire detection accuracy of the competing methods will be evaluated in terms of $\textbf{Precision (P)}$, $\textbf{Recall (R)}$ and $\textbf{F1-Score (F)}$. To formalize these metrics, it is necessary to define the sets of $\textbf{True Positive (TP)}$, $\textbf{False Positive (FP)}$ and $\textbf{False Negative (FN)}$. Our test set contains both positive and negative samples, i.e. video depicting fire events or not, respectively. Each positive video shows one fire event only. Each prediction is evaluated by comparing the fire start instant $\mathbf{g}$ with the detection istant $\mathbf{p}$. We indicate as the fire start instant the first frame in which it is visible. It is manually labelled by the human operator after visioning the whole video thus it is a ground truth. Notwithstanding this, we consider that automatic methods could anticipate human detection by a few seconds, namely $ \mathbf{\Delta}_{\text{t}} = 5\ $seconds. At the same time, we discard detections occurring later than $\mathbf{T}_{\text{max}} = 60$ seconds. Consequently, we define:

• $\textbf{TP}$: all the detections occurring in positive videos at $ \mathbf{\boldsymbol{g - \Delta t \leq p \leq g + T_{\text{max}}}}$;
• $\textbf{FP}$: all the detections occurring at any $\mathbf{t}$ in $\mathbf{r}$ in positive or negative videos at $\mathbf{\boldsymbol{p < \max(0, g - \Delta t)}}$;
• $\textbf{FN}$: the set of positive videos for which no fire detection occurs;

Defined these sets for $\textbf{TP}$, $\textbf{FP}$ and $\textbf{FN}$, we can compute the $\textbf{P}$, $\textbf{R}$ and $\textbf{F}$ with respect to the number of $\textbf{|TP|}$, $\textbf{|FP|}$ and $\textbf{|FN|}$:

$\mathbf{\boldsymbol{P = \frac{|TP|}{|TP| + |FP|}}}$

$\mathbf{\boldsymbol{R = \frac{|TP|}{|TP| + |FN|}}}$

$\mathbf{\boldsymbol{F = \frac{2 \cdot (P \cdot R)}{P + R}}}$

• $\textbf{P}$ assumes values in the range $\mathbf{[0, 1]}$ and measures the capability of the methods to reject $\textbf{FP}$; the higher is $\textbf{P}$, the higher the specificity of the method;
• $\textbf{R}$ assumes values in the range $\mathbf{[0, 1]}$ and evaluates the sensitivity of the method to detect fire; the higher is $\textbf{R}$, the higher the sensitivity of the method;
• $\textbf{F}$ assumes values from $\mathbf{[0, 1]}$ and measures the balance between $\textbf{P}$ and $\textbf{R}$;

Then, we compute the $\textbf{Processing Frame Rate (PFR)}$, namely the average number of frames processed by the method in one second on a target $\textbf{GPU}$. In particular, being $\mathbf{N}$ the total number of frames processed in the test set and $\mathbf{t}_{\text{i}}$ the processing time in seconds for a single frame, the $\textbf{PFR}$ is computed as follows:

$\mathbf{\boldsymbol{PFR = \frac{1}{\frac{\sum_{i=1}^{N}{t_i}}{N}}}}$

The higher is $\mathbf{PFR}$, higher is the processing speed of the method. To normalize this value with respect to the minimum processing frame rate needed to achieve real-time performance, namely $\mathbf{PFR}_{\text{target}}$, we compute the $\mathbf{PFR}_{\text{delta}}$ score as follows:

$\mathbf{PFR_{\boldsymbol{delta}} = \max(0, \frac{PFR_{\boldsymbol{target}}}{PFR} - 1)}$

Then, We measure the $\textbf{Memory Usage (MEM)}$, namely the memory in GB occupied by the method on the target $\textbf{GPU}$. The lower is $\textbf{MEM}$, lower is the necessary memory on the processing device. To normalize this value with respect to the maximum $\textbf{GPU}$ memory available on the target processing device, namely $\mathbf{MEM}_{\text{target}}$, we compute the $\mathbf{MEM}_{\text{delta}}$ score as follows:

$\mathbf{MEM_{\boldsymbol{delta}} = \max(0, \frac{MEM}{MEM_{\boldsymbol{target}}} - 1)}$

Finally, we define the $\textbf{Constrained Fire Detection Score (CFDS)}$ computed as follows:

$\mathbf{CFDS = \frac{F}{(1 + PFR_{\boldsymbol{delta}}) \cdot (1 + MEM_{\boldsymbol{delta}})}}$

The method achieving the highest $\textbf{CFDS}$ will be crowned the winner of the ONFIRE 2025 Contest, showcasing the best balance between fire detection accuracy across a variety of scenarios with differing levels of difficulty, promptness in notification and resource efficiency. The winners will be those securing top performance in the scenario-specific rankings and the overall ranking.

Rules

1. The deadline for the submission of the methods is 6th June, 2025. The submission must be done with an email in which the participants share (directly or with external links) the trained model, the code and the report. Please follow the detailed instructions reported here.
2. The participants can receive the training set and its annotations by sending an email, in which they also communicate the name of the team.
3. The participants can use these training samples and annotations, but also additional videos.
4. The participants must submit their trained model and their code by carefully following the detailed instructions reported here.
5. The participants are strongly encouraged to submit a contest paper by the deadline of 13th June, 2025. The paper can be submitted through Easychair. Authors can find complete instructions of how to format their papers here. The maximum number of pages is 12 including references. Accepted papers will be included in the ICIAP 2025 Workshops Proceedings.

Instructions

The methods proposed by the participants will be executed on a private test set. To leave the participants totally free to use all the software libraries they prefer and to correctly reproduce their processing pipeline, the evaluation will be done on Google Colab (follow this tutorial) by running the code submitted by the participants on the samples of our private test set.

Therefore, the participants must submit an archive (download an example) including the following elements:

• A Python script test.py, which takes as input the folder of the test videos (--videos) and produces as output a TXT file with the detection time (or nothing) for each video in a folder (--results). Thus, the script may be executed with the following command:
  python test.py --videos foo_videos/ --results foo_results/
• A Google Colab Notebook test.ipynb, which includes the commands for installing all the software requirements and executes the script test.py.
• All the files necessary for running the test, namely the trained model, additional scripts and so on.

The provided sample test.py also includes the reading of the TXT files with the annotations. Each file includes the fire ignition instant or nothing if it is a negative video. The files of the results will be formatted exactly in the same way. The provided sample test.py includes the writing of the files with the results.

The submission must be done by email. The archive file can be attached to the e-mail or shared with external links. We strongly recommend to follow the example of code to prepare the submission.