International Planning Competition 2023 HTN Tracks

IPC 2023 HTN Tracks

International Planning Competition 2023 HTN Tracks

This is the website for the hierarchical (HTN) tracks of the IPC 2023. It is the 2nd IPC containing hierarchical tracks, after its first occurance in 2020.

Calls

Please forward the following calls to all interested parties.

Schedule

Event Date
Call for domains October 12, 2022
Call for participation October, 2022
Domain expression of interest deadline December 31, 2022
Domain submission deadline February 28, 2023
Demo problems provided February 10, 2023
Initial planner submission February 28, 2023
Feature stop (final planner submission) May 31, 2023, AoE
Planner Abstract submission deadline May 31, 2023, AoE
Contest run June, 2023
Results announced July 12, 2023
Result analysis deadline August, 2023

Tracks

There will be (up to, provided sufficient interest) six tracks in IPC’s HTN track. These are:

The semantics of these tracks is explained in the following.

Total-Order

Partial-Order

Agile Track

Satisficing Track

Optimal Track

HDDL Fragment

We will use the same input language as the previous HTN IPC 2020

Technical Issues.

Competitors are required to submit their code via a github repository that was set up by the track’s organisers. At the time of submission, these repositories will be private and will be made public after the competition results have been announced.

Until the feature stop (see Schedule) competitors can make any change to their planners. After the feature stop, competitors are only allowed to send pull requests with bug fixes. We will review every pull request with its accompanying description of the bug fix to make sure that no significant changes or parameter tuning are possible.

As in 2020, we will provide a continous integration system. We will run planners on sample problems and publish the resulting outputs. To faciliate the detection of anomalies, each competitor is required to also submit a script that validates the planners output and that is responsible to highlight (potential) errors that need to be investigated.

As in the HTN IPC of 2020, we will use the container software Apptainer (formerly known as Singularity) to promote reproducibility and simplify program compilation. You can find the documentation on how the Apptainer files are written here.

We will automatically extract the configurations you want to submit from your repository. We create one configuration per Apptainer file in the root directory of your repository. Apptainer files are files whose file ending is .def.

Registration

In order to participate in the HTN IPC, you need to register until February 28, 2023. To do so, please send an email to Gregor Behnke containing the following information

As the other IPC tracks, the HTN track allows for multiple submissions to use the same planner codebase. These submissions can, e.g., differ in the configuration of the planner such as the used heuristic(s) or search techniques. Each of these configurations will (most likely) be a separate apptainer definition file (formerly Singularity file).

The number of submissions per participant (natural person who participates) is limited to three per track. That means that any person may only be named as a “participant” for at most three submission per track. This limit applies to the number of configurations and not codebases. I.e. the limit is already reached if a participant submits one codebase that is used for three configurations in one track.

After some discussion we have decided to update and clarify the submission rules as follows (on 24-02-2023):

Authors are allowed to submit an arbitrary number – within reason – of different planning systems per track. Two planning systems are considered different if the relevant codebase, i.e., the parts of the code that are actually being executed, differs substantially, as judged by the organizers. (This allows, e.g., for submitting different planning approaches as separate planning systems even if they are part of a common software.)

For each planning system submitted, a maximum of three different configurations may be submitted.

A modification of somebody else’s HTN planning system is a valid submission if and only if the original author(s) have been made aware of the submission and are consenting.

Participants

The following planning systems participated in the HTN track of the IPC 2023. In this table, you can find the link to the repositories for each competitior as well as links to the apptainer containers we build based on them.

Participants

In the following table, you can find all participating planners from the HTN track of the IPC 2023. The linked github repositories contain the planners as used in the IPC evaluation. We also provide the apptainer images we build for every planner. Note: these links are currently not available as we have not uploaded the files yet Note: We cannot provide the apptainer containers for the configurations ppro-xo-opt-as-dof-ao, ppro-xo-opt-as-dof-lmc, and ppro-xo-opt-as-lmc. These require CPLEX to build – which cannot be provided as public download. If you which to run these planners, you need to obtain your own license for CPLEX and add CPLEX to the build process of the repository.

Participant Authors Github Containers
Aries Arthur Bit-Monnot Aries aries-sat
aries-agile
aries-opt
SIADEX Ignacio Vellido Expósito, Jorge Soler-Padial,
Juan Fernández Olivares, Luis Castillo
SIADEX SIADEX
Lifted Linear Ying Xian Wu, Conny Olz,
Songtuan Lin, Pascal Bercher
Lifted Linear LiftedLinear-1
LiftedLinear-2
LiftedLinear-3
Linear Simple Ying Xian Wu, Conny Olz,
Songtuan Lin, Pascal Bercher
Linear Simple LinearSimple-agile-1
LinearSimple-agile-2
LinearSimple-agile-3
LinearSimple-sat-1
LinearSimple-sat-2
LinearSimple-sat-3
Linear Complex Ying Xian Wu, Conny Olz,
Songtuan Lin, Pascal Bercher
Linear Complex LinearComplex-agile-1
LinearComplex-agile-2
LinearComplex-agile-3
LinearComplex-sat-1
LinearComplex-sat-2
LinearComplex-sat-3
PANDApro Daniel Höller PANDApro lamda-po-agl-gas-ao
lamda-po-agl-gas-lmc
lamda-to-agl-gbfs-ao
lamda-to-agl-gbfs-lmc
ppro-po-agl-gas-ff
ppro-po-sat-gas-ff
ppro-to-agl-gbfs-add
ppro-to-sat-gbfs-add
ppro-xo-opt-as-dof-ao (requires CPLEX)
ppro-xo-opt-as-dof-lmc (requires CPLEX)
ppro-xo-opt-as-lmc (requires CPLEX)
TOAD Daniel Höller TOAD toad-io-dfad
toad-io-ff
toad-po-dfad
PandaDealer Conny Olz, Daniel Höller,
Pascal Bercher
PandaDealer PandaDealer-agile-1
PandaDealer-agile-2
PandaDealer-agile-lama
PandaDealer-optimal
PandaDealer-sat-1
PandaDealer-sat-2
PDDL4J Gaspard Quenard, Oleksandr Firsov,
Damien Pellier, Humbert Fiorino
PDDL4J LiftedTreePath
OptiPlan

Current Status of Planner Submissions

See status page.

Benchmark Set

We have used most of the domain from the previous IPC 2020. For both the Total-Order and the Partial-Order Tracks, we have added two new domains:

The full set of instances used for the HTN Track of the IPC 2023 is available on github.

Results

The IPC 2023 has concluded and the results have been presented at ICAPS 2023 in Prague. The results presentation can be found here. The full results tables can be found here. The results website also contains links to CSV files of the results (and additionally coverage data).

You can download the raw log files of the planners and the per-instances scores for the TO Track and the PO Track.

The winners are:

Track Winner Runner-Up
Total-Order Satisficing PandaDealer-agile-lama
Conny Olz, Daniel Höller, Pascal Bercher
PANDApro $\lambda$ AO
Daniel Höller
Total-Order Agile PandaDealer-agile-1 and PandaDealer-agile-lama
Conny Olz, Daniel Höller, Pascal Bercher
PANDApro $\lambda$ lm-cut
Daniel Höller
Total-Order Optimal PandaDealer-optimal
Conny Olz, Daniel Höller, Pascal Bercher
PANDApro lm-cut and PANDApro DOF
Daniel Höller
Partial-Order Satisficing Grounded-Linear-Satisficing
Ying Xian Wu, Conny Olz, Songtuan Lin, Pascal Bercher
PANDApro $\lambda$ AO and lm-cut
Daniel Höller
Partial-Order Agile Grounded-Linear-Agile
Ying Xian Wu, Conny Olz, Songtuan Lin, Pascal Bercher
PANDApro $\lambda$ AO and lm-cut
Daniel Höller
Partial-Order Optimal PANDApro lm-cut
Daniel Höller
Aries
Arthur Bit-Monnot

The full results tables can be found here.

Proceedings and Planner Abstracts

The preliminary versions of the planner abstracts can be found here. The participants can still submit a final version, which can also include an analysis of the behaviour of their planner in the IPC. We will publish these proceedings once we have them.

Participant Planner Abstract
Aries Aries
SIADEX SIADEX
SIADEX is the only planner that participated with exactly the same version as in the IPC2020.
We thus refer to the paper from the IPC2020
Lifted Linear Lifted Linear
Linear Simple Linear Simple
Linear Complex Linear Complex
PANDApro PANDApro
PANDApro $\lambda$
TOAD TOAD
PandaDealer PandaDealer
PDDL4J LiftedTreePath
OptiPlan

Organizers

Contact us: ipc2023-htn@googlegroups.com