Exploring Our Focus
The primary aim of this workshop is to provide practical examples, case studies and guidance towards a comprehensive understanding of reproducibility and objective performance evaluation in intelligent robotics, through expert inspiration and mentoring, presentations of concrete examples of R- and r-articles, benchmarking procedures and interactive discussion. By spotlighting the significance of consistent and reproducible outcomes in real-world robotic applications, we aim to push the boundaries of knowledge, encourage robust discussions, and pave the way for collaborative research that will define the next era of robotic advancement.
Beside general aspects about reproducibility and benchmarking in robotic research, the workshop will particularly focus on the common challenges and possible solutions related to autonomous mobile systems, manipulation and grasping and humanoids using R- and r-articles as key tools for advancing reproducibility and objective quantitative evaluation of results and to support abstraction of general models applicable to all intelligent robots. This workshop is part of a series intertwines with Roboracer/F1Tenth competition that has been organized since 2016 at various conferences, focusing on scaled autonomous vehicle racing, with an emphasis on benchmarking/reproducibility at three different levels: pre-field simulation, simulation-to-real transfer and real-world experiments. Indeed, we will try to engage people from all the robotics competitions, both organizers and participants, to share their experience, because they are strongly aware of the reproducibility issue and they always seek consistent methodologies to measure and compare results obtained by the competing teams. Their perspectives will bring a precious contribution to our workshop: similar problems in many diverse robotics and environmental setups will expand the insight about reproducibility and benchmarking, and will help to end up with some general considerations, standard best practices and possible advancements and new ideas to be applied to the majority of experimental robotics subfields.
This workshop can provide valuable inputs and methodologies to the post-processing data analyzing and benchmarking of obtained performance, while in turn the competition can feed new challenges and aspects to be discussed within the community in this and future events.
Relevance and Expected Impact:
Robotics has made tremendous progress. However, the technology readiness and transition into everyday life is often less quick than expected due to lacking procedures and standards to benchmark and assure safe behaviors of robots. This workshop faces such topics, focusing on the critical aspect of reproducibility in general robotics and, specifically, in the critical area of research on autonomous locomotion as specific case study of interest.
The introduction and adoption of R- and r-articles represent a significant advancement in this effort. R-articles, with their emphasis on the reproducibility of results, provide a solid foundation for credible peer evaluations and objective assessments of system behaviors. By requiring the provision of all information necessary to replicate experiments, R-articles ensure that research can be independently verified, thus enhancing the reliability and credibility of the findings. The subsequent creation of r-articles, which detail attempts to reproduce the experiments, further solidifies this process, allowing for the confirmation or refutation of original results. This approach has the potential to revolutionize research in robotics by establishing a culture of transparency and replicability, which are essential for the field's advancement.
The appreciation of the significance of these topics is widespread within the community, especially as society progresses towards a future where robots play an integral role in diverse sectors, from industry to healthcare, research, civilian and everyday life. By emphasizing state-of-the-art techniques and fostering discussions on challenges, the workshop is poised to stand as a landmark of originality and diversity for IROS 2025. Furthermore, this event benefits from a series of previous workshops dedicated to the same topic, aiming to create i) a continuous dialogue and knowledge evolution in the field; ii) benefits and mutual exchanges from/to competitions on autonomous robots, highlighting with practical demonstration at field the great importance and huge reach that the reproducibility topic assumes nowadays.
Delving into Key Topics
This workshop is designed to encompass a wide range of topics, with special focus, but not limited, to the following areas:
• Benchmarking methods for robotics research
• R-, r- and reply article in Robotics
• Best Practices and deployment studies for Autonomous mobile robots
• Software Tools and Platforms
• Benchmarking methods for robotics research
• Statistical significance of results
• Benchmarking of Machine Learning and Deep Learning systems used in intelligent robotics
• Benchmarking humanoids
• Open-Source and Novel Platforms for Reproducible Research
• Robotic Task Standardization
• Peer Evaluation and Metrics
• Real-World Case Studies