Define the task

Before writing any code, it’s essential to carefully plan and document your task to ensure its relevance, feasibility, and compatibility with the OpenProblems framework.

Create a README

Begin by creating a README file at src/<task_id>/README.md with the following sections.

Task motivation

Start by explaining the motivation behind your proposed task. Describe the biological or computational problem you aim to address and why it’s important. Discuss the current state of research in this area and any gaps or challenges that your task could help address. This section should convince readers of the significance and relevance of your task.

Task description

Provide a clear and concise description of your task, detailing the specific problem it aims to solve. Outline the input data types, the expected output, and any assumptions or constraints. Be sure to explain any terminology or concepts that are essential for understanding the task.

Proposed ground-truth in datasets

Describe the datasets you plan to use for your task. OpenProblems offers a standard set of datasets (See “Common datasets”) which you can peruse through.

Explain how these datasets will provide the ground-truth for evaluating the methods implemented in your task. If possible, include references or links to the datasets to facilitate reproducibility.

Initial set of nethods to implement

List the initial set of methods you plan to implement for your task. Briefly describe each method’s core ideas and algorithms, and explain why you think they are suitable for your task.

Consider including both established and cutting-edge methods to provide a comprehensive benchmarking of the state-of-the-art.

Proposed control methods

Outline the control (or baseline) methods you propose for your task. These methods serve as a starting point to test the relative accuracy of new methods in the task and as quality control for the defined metrics.

Include both positive controls, which are methods with known outcomes resulting in the best possible metric values, and negative controls, which are simple, naive, or random methods that do not rely on sophisticated techniques or domain knowledge. Explain the rationale for your chosen controls.

Proposed Metrics

Describe the metrics you propose for evaluating the performance of methods in your task. Explain the rationale for selecting these metrics and how they will accurately assess the methods’ success in addressing the task’s challenges. Consider including multiple metrics to capture different aspects of method performance.

Create a GitHub issue

Once you have completed your README file, share it with the OpenProblems community for feedback and discussion by creating an issue on our issue tracker.

This collaborative process will help ensure that your task is well-defined, relevant, and compatible with the OpenProblems framework. Additionally, it informs others of your ongoing work in this area and establishes a foundation for potential collaboration.

Next steps

Based on the information contained in the README, you are now well-equipped to begin designing the API for the new benchmarking task.