Real-Life Experiences in Using Open Source for Autonomy Applications ^†

Abstract

The open-source paradigm is well known for its unique potential to crowd-source innovation with spectacular examples of success, such as Linux. The knowledge base of methods to build more successful open-source communities has been growing with concepts such as independent governance, community building, and funding mechanisms. In mobility, an open-source autonomy stack called Autoware was the basis of a significant public transportation research project called ISEAUTO, and a companion project, PolyVerif, focused on building validation and verification technologies. This paper outlines the real-life experiences of two teams using the Autoware framework, providing lessons and recommendations for improving open-source development.

1. Introduction

Open-source systems have gained great traction, especially with projects with significant software content in information technology. Conventionally, an individual or a company invests in a technology area, owns all rights to it, protects the generated intellectual property, and monetizes this investment through customer sales. In an open-source paradigm, a “community” structure is constructed where many participants can contribute, and all can use the collective intellectual property. Open-source environments are especially effective in situations where there are a large number of potential suppliers and a large number of potential use models. The open-source environment allows an incremental contributor to leverage the massive amount of work of others and contribute something valuable to the whole. When working well, the amount of accumulated IP and capability can be breathtakingly large, and the rate of innovation accelerates. The model instantly democratizes technology with the consequence that monetization and differentiation must be built in other ways.

As outlined in “Unsettled Issues Regarding Autonomous Vehicles and Open-source Software” [1], an automobile has become software on wheels, and, indeed, there has been a great deal of progress for open-source projects within the automobile software stack with projects such as Automotive Grade Linux and Android-Auto. Not surprisingly, the progress has been largely on the “IT” or infotainment part of the automotive hardware/software stack. For safety-critical systems with a special emphasis on autonomy, the software system is dominated by proprietary solutions. However, as outlined in “Unsettled Technology Areas in Autonomous Vehicle Test and Validation” [2], significant innovation is required to design and validate autonomy-based systems, so there is large motivation to build open-source projects to accelerate the State-of-the-Art for autonomy.

With this in mind, an open-source autonomy project, Autoware, was launched in 2015 [3,4]. Autoware can be considered an operating system for autonomous vehicles (AVs) containing all the major components (localization, mapping, route planning, vehicle/pedestrian identification, tracking, simulation, lane detection, sensor drivers/fusion, etc.). Soon after its initial release, the industry demand was too intense to be handled by an academic research group. Therefore, in December 2015, Tier IV, a private company, was formed to maintain and support Autoware. As the interest in collaboration grew and governance issues became more relevant, Tier IV transferred its assets to The Autoware Foundation (nonprofit) in December 2018 in an open-source structure. Autoware uses a standard Apache 2.0 open-source licensing structure. The Autoware Foundation also boasts over 50 members with a mix of semiconductors (e.g., NXP), electronic device automation (e.g., Cadence), sensor companies (e.g., Velodyne), and academia (e.g., Tallinn University of Technology (TalTech)).

After its first release, the Autoware team spent much effort picking used models of interest while trying to maintain the older version. At some point, this strategy was not viable, and the Autoware team moved to a newer version called “Autoware.AI”, which introduced improvements in perception algorithms, sensor fusion, and mapping capabilities. It aimed to enhance self-driving vehicles’ autonomy and safety by incorporating State-of-the-Art technologies and methodologies. Subsequent versions, such as “Autoware.universe”, took a step further by adopting a modular and scalable architecture. This version aimed to provide a more flexible platform for developers, allowing them to integrate and customize modules based on their specific requirements easily. “Autoware.universe” facilitated the development of additional packages that can be layered on top of Autoware Core, extending its functionality. The goal was to build a modular approach that allowed for the seamless integration of advanced research and development, enabling the early experimentation and deployment of capabilities to be integrated into forthcoming versions of Autoware Core. Autoware Universe allows for applying Autoware to intricate and demanding scenarios, such as Robo-Taxis operating in dense urban environments or high-speed autonomous racing [5]. Compared to other open-source platforms such as Apollo [6], Autoware offers a more modular approach, facilitating easier adaptation to various autonomous vehicle applications. While Apollo integrates more proprietary code, Autoware’s open governance allows for greater community involvement and broader contributions to its ecosystem.

Starting in 2017, two major projects, ISEAUTO and PolyVerif, used Autoware and associated ecosystem partners (simulators) as foundational pieces to build broader and deeper solutions. In the remainder of this paper, we will outline our experience using Autoware and generalize more broadly to provide recommendations for open-source environments.

2. ISEAUTO Project

In 2017, TalTech’s AV research lab initiated a self-driving vehicle project (ISEAUTO), driven by innovation and the university’s impending centenary. Given the challenging timelines and resource constraints, the TalTech team leveraged the Autoware AV framework. The result in [7] was a functional vehicle within one year. The project generated enormous interest for the TalTech team with research funding for many pilots and engagements from larger European Union projects such as Fabulos [8] and Sohjoa [9].

From a development point of view, ISEAUTO moved from a small research project to a multi-year research effort and development team with ongoing development commitments and significant investments from several PhD students. The incremental work was built on top of the Autoware AV stack. While the AV stack, like Apollo, offers a hybrid approach of open-source and proprietary elements, our focus on Autoware provides insights into the unique challenges posed by fully open-source platforms. The Autoware Foundation, responsible for its development, transitioned from “Autoware.ai” to “Autoware.auto” and, currently, to “Autoware.universe”. Each iteration of the core platform brought its dependencies and requirements, rendering the migration process between them a complex and challenging endeavor for the ISEAUTO team.

Furthermore, as the core architecture of Autoware evolved, its crucial open-source modules for localization, detection, and planning also underwent significant changes. These shifts added complexity to development, requiring teams to adapt their codebases continually. The internal changes in Autoware’s structure compounded these challenges, highlighting the difficulty of maintaining compatibility with open-source AV software. As the Autoware Foundation advances with Autoware.universe, developers must navigate core platform shifts and module changes, emphasizing the need for a robust adaptation strategy. It is estimated that ISEAUTO will need at least six months to transition to Autoware.universe.

2.1. Operating System Transition

Robotic Operating System (ROS), the base platform for Autoware, also experienced several notable shifts between different distributions and versions. While both Kinetic and Melodic are ROS 1 distributions, the upgrade from Kinetic to Melodic required extensive effort from ISEAUTO developers. Package modifications and upgrades were required to ensure compatibility with the nuances of each distribution. These ROS 1 distribution transitions posed unique challenges, impacting the development timeline and resource allocation. They also served as precursors to the deeper challenges encountered during the subsequent shift to ROS 2, further highlighting the intricate nature of managing open-source software evolution in AV development. This transition brought fundamental changes, which required teams to invest considerable time and resources in migrating their projects and packages. The challenges extend beyond code adaptation, encompassing potential hardware and middleware compatibility issues. This version shift exemplifies how open-source software evolution may demand substantial efforts and cause temporary disruptions in ongoing projects.

2.2. AV Simulation Platform Sunset

A software testing toolkit is one of the essential requirements for AV development. SVL [10] was a project sponsored by LG as a key partner to the overall Autoware Project. In 2022, the platform was discontinued. The platform’s discontinuation left development teams and researchers in a precarious position. Teams heavily reliant on SVL faced the sudden loss of a critical tool, requiring them to search and invest in alternative solutions such as the CARLA simulator [11] or, in some cases, develop an in-house simulator. In the ISEAUTO case, the validation platform [12] had to be upgraded, and its elements, including the 3D virtual environment, vehicle model, and scenario generation scripts, were transferred to the new CARLA simulator based on a different game engine. It took nearly a year for the TalTech AV research group team to complete the transfer of all features. The unexpected withdrawal of support for a vital component jeopardized ongoing research and development and underscored the risks associated with relying solely on open-source platforms.

The dynamism of these platforms means that a project’s success is intricately tied to the sustainability and longevity of the underlying software. Teams investing time and effort into an open-source solution jeopardize their work if the platform loses momentum or support, or if it discontinues. This precarious situation can lead to wasted efforts and stalled research and development initiatives. The extra efforts required to adapt to evolving open-source software and the uncertainties surrounding platform longevity emphasize the need for strategic planning and risk mitigation in AV development.

3. PolyVerif

Validation and Verification (V&V) of AI-based cyber-physical systems is a vexing problem with limited signs of significant progress. In the 2019 timeframe, the team from Florida PolyTechnic University introduced PolyVerif [13]. PolyVerif provided an AI design and verification flow consisting of the digital twin creation process, an open-source AV engine, access to several open-source physics-based simulators, symbolic test generation engine, example test scenarios from documented AV accidents, and native design-for-experiment layer for each of the major subsystems of an AV (detection, perception, localization, and path planning). PolyVerif’s objective was to arm V&V researchers to perform research that extends the State-of-the-Art on any of the major axes of interest and use the remainder of the infrastructure to demonstrate the solution’s viability quickly. Given its open-source nature, researchers can contribute their innovations to the project. Using this critical property of open-source environments, the innovative potential of the whole research community to solve these vexing issues is greatly enhanced. Figure 1 below shows the PolyVerif flow.

The PolyVerif platform is developed using open-source technologies to validate AV stack modules, viz LGSVL, SCENIC [14], and Autoware [15,16]. It uses an LGSVL simulator to simulate the environment and sensors and is tested with the Autoware AV stack. Various logging modules for ground truth data collection and comparator algorithms to generate validation metrics were developed for the PolyVerif validation suite.

During the course of development, LGSVL open-source support was stopped, so other simulators were investigated. Fortunately, LGSVL was further supported and taken ahead by another open-source group—Open Source Self Driving Car (OSSDC) Initiative Simulator workgroup. PolyVerif was eventually migrated to the new simulator with some changes in the framework to support the complete validation functionality. The server dependencies were removed to facilitate local usage by consuming teams. Moving forward as part of the roadmap and to enable long-term support, the PolyVerif team is migrating to Autoware.Universe and the Autoware simulator (AWSIM).

PolyVerif migration to use the AWSIM is approximately four months with four resources, i.e., 16 man-months of effort, working full time, including development, migration, and testing. This work involved understanding the new architecture and functionality and migrating the plugins and algorithms to the new architecture. It required adding plugins to the existing architecture for ground truth data logging and integration with all the API’s. Moreover, additional efforts are needed to create test scenarios in the OpenSCENARIO language, as Autoware.universe supports only OpenSCENARIO as the Scene description language. All the Test scenario scripts previously written in SCENIC and Python must also be migrated to OpenSCENARIO.

4. Recommendations

For the ISEAUTO and PolyVerif teams, the live experience of working with the active Autoware project provided insights into the mechanisms that worked and places for improvement. As outlined in “Unsettled Issues Regarding AVs and Open-source Software”, Autoware used most of the critical elements of a successful open-source paradigm (Governance, Licensing, Active producers and consumers, crowd-sourced innovation, and motivations for contributors).

4.1. Abstract Interface

Autoware, like many programming projects, is focused on developing core intellectual property (in this case, software). This focus creates a high energy and rapid, organic growth of the software base. From the consumer’s perspective of this IP, it is positive because of the capability that comes online, but it also creates a large maintenance debt for code dependent on older versions of the software. In the world of computers, the notion of an Instruction Set Architecture (ISA) was seminal in the birth of the modern computer industry. With an ISA, software and hardware teams could innovate separately with the assurance that the ISA would safely preserve their development.

Even more important than code, open-source projects should focus on developing abstract interfaces and then define their permanence so that external users can appropriately manage dependencies. The use of abstract interfaces allows for an orderly transition of software and has the interesting side effect of making the project more approachable to new partners. The organic development of software tends to grow in complexity over time, with the organization largely determined by historical transitions. With Autoware, bringing new people on board has become increasingly more difficult because of the ramp-up time required in the code stream. Coherent structured interfaces greatly simplify this task as well.

4.2. Support

For any consumer of open-source IP, the intent and current status of the project need to be communicated in several dimensions. We recommend a chart as in

Table 1. Levels of communication for open-source IP.

Level	Openness	Maturity	Churn	Ecosystem
L0	All Proprietary	One Case Works	High Degree of Churn	None
L1	Mostly Proprietary	Hundreds of Cases Work	Some Stable Interfaces and Core	One Research Group
L2	Majority Proprietary	Product Validation Flow Available	Interfaces Stable, but Core Has Churn	Multiple Research Groups
L3	Majority Open	Third-Party Testing Available	Interface Stable, Most of the Code Stable	Some Components Professionally Supported and a Driving User
L4	Mostly Open	Small Number of Production Customers	Interfaces and Core Code Stable over a Year	Most Components Professionally Supported and Some Users
L5	All Open	Large Number of Production	Stable Interfaces, Stable Core over	Professional Support Available

Openness: For consumers of open-source projects, it is vital to understand the extent to which the project is truly open-source. Autoware was at level 5 in the case of autonomy, while some others advertised open-source projects, such as Apollo, which had a mix of open-source and proprietary code from Baidu; Maturity: In applications such as automotive, there are significant requirements around safety and cybersecurity. Thus, understanding the maturity of the open-source components is vital, as is the nature of the validation suite used to validate code contributions. Further, compliance with these standards must be communicated when safety standards are involved (e.g., ISO 26262); Churn: In abstract interfaces and code, it is important to understand clear metrics around churn. These metrics can come from the code base, and an indicator of the impact of the churn is the number of software production consumers; Ecosystem: Finally, a clear and crisp understanding of the ecosystem from sub-suppliers, service providers, and even regulators is very useful.

with focus points on openness, maturity, churn, and ecosystem development.

5. Sustainability

How does one manage an open-source project without funding? Someone must manage the infrastructure, and, most importantly, there must be a method for validation of contributions. The first method for monetizing open-source systems was demonstrated in the late 1980s by Kaz Hashimoto, who was able to sell the services and maintenance model with Cygnus Solutions (predecessor to IBM Redhat). Others, such as Google, have perfected the enabled ecosystem model where the open-source structure provides enablement for other profit-making business models.

For any consumers of open-source environments, it is very important to understand the answers to the sustainability question. It should be noted that lower scores in Table 1 are not “bad”; however, they guide the level of investment, which is prudent with open-source project dependencies. This communication around sustainability also provides the motivation to invest in and support the core project before further investments above the core.

6. Conclusions

Modern open-source communities are cognizant of State-of-the-Art techniques such as independent governance structures, explicit methods for community building, and independent funding mechanisms to build viable sustainable structures. With four years of experience in consuming open-source code in the form of the Autoware project, the ISEAUTO and Polyvore teams have gained insight into the advantages and disadvantages of this paradigm. The disadvantages center around the constant accumulation of maintenance debt, the potential for obsolescence, and clear communication of intent. While these issues are difficult in all software domains, they are even more acute in fields with heavy safety elements such as autonomy. In this paper, the teams propose further methods and meta-information, which can move the State-of-the-Art of open-source projects forward.