定义组件的临时性能指标#
定义临时性能指标的动机#
页面目标#
本页介绍了用于定义指标的策略,以评估 Autoware 组件的临时性能.术语 temporal performance 经常在整个页面中使用,以区分函数性能(也称为准确性)和与时间相关的性能.
预计 Autoware 采用的大多数算法都以尽可能高的频率和较短的响应时间执行.为了实现安全的自动驾驶,期望的结果之一是感知和实际情况之间没有时间差.时间间隔通常称为延迟.如果延迟很大,系统可能会根据过时的情况确定轨迹和机动.因此,如果由于延迟导致实际情况与感知情况不同,系统可能会做出意想不到的决定.
如上所述,本页介绍了用于定义量度的策略.此外,该页面还包含对 Autoware 的主要功能至关重要的示例指标列表：定位(Localization)、感知、规划和控制.
注意
其他功能,例如用于诊断的系统组件,目前被排除在外.但是,在不久的将来,它们将被考虑在内.
临时性能指标的贡献#
临时性能指标对于评估 Autoware 非常重要.这些指标对于评估新算法和逻辑导致的延迟特别有用.在车辆集成阶段,当比较台式计算机上软件和车辆上的软件的时间性能时,可以使用它们.
此外,这些指标对于中间件、作系统和计算机的设计人员和评估者也很有用.它们是根据用户和产品要求选择的.其中一项要求是提供足够的临时性能来执行 Autoware. 足够的临时性能 被定义为临时性能要求,但定义该要求可能具有挑战性,因为它因产品类型、运营设计域 (ODD) 和其他因素而异.然后,本页特别关注临时性能指标,而不是需求.
临时性能指标对于评估 Autoware 的可靠性非常重要.但是,确保 Autoware 的可靠性不仅需要考虑临时性能指标,还需要考虑其他指标.
用于评估指标的工具#
有多种工具可用于根据页面中列出的指标评估 Autoware.例如,在评估 Linux 和 ROS 2 上的 Autoware 时,CARET 和 ros2_tracing 都是推荐选项.如果要使用这些工具中的任何一个来测量量度,请参阅相应的用户指南以获取说明.请务必注意,如果您将 Autoware 导入到 Linux 和 ROS 2 以外的平台,则需要选择受支持的工具进行评估.
该页面并非旨在提供有关如何使用这些工具或测量指标的说明.它的主要关注点是指标本身,因为它们比使用的特定工具更重要.无论采用何种平台,这些量度都保持其相关性.
用于定义临时性能指标的策略#
如上所述,Autoware 的配置因产品类型、ODD 和其他因素而异.配置的多样性使得很难定义用于评估 Autoware 的统一指标. 但是,即使配置发生更改,用于定义它们的策略基本上也会被重用.每个临时性能指标都分为两种类型：执行频率和响应时间.尽管有许多类型的指标,例如通信延迟,但为了简单起见,我们只考虑了两种类型. 使用进程间通信 (IPC) 消息的速率来观察执行频率.您会在 Autoware 中找到大量消息,但您不必处理所有消息.某些消息可能对功能至关重要,应选择它们进行评估. 响应时间是经过一系列处理所经过的持续时间.一系列处理称为路径.响应时间是根据路径的开始和结束的时间戳计算的.尽管在 Autoware 中可以定义许多路径,但您必须选择重要的路径.
作为提示,以下是 message 和 path 的一些特征,以便选择量度.
- 使用传感器观测值的边界上的消息和路径
- 功能边界上的信息和路径,例如,感知和规划的边界
- 基于定时器的频率切换的边界上的消息和路径
- 同步和合并两条不同消息的边界上的消息和路径
- 必须以预期频率传输的消息,例如车辆命令消息
这些提示对大多数配置都有帮助,但可能存在例外情况.精确定义指标需要了解配置.
此外,建议从架构级别到详细设计和实现级别逐步确定指标.混合不同粒度级别的指标可能会令人困惑.
示例量度列表#
本节演示了如何根据解释的策略定义指标,并提供了根据 教程.该部分分为多个小节,每个小节都包含一个模型图和一个随附的列表,用于解释重要的临时性能指标.每个模型都配备了检查点,用作这些指标的指标.
第一小节介绍了顶级时间性能指标,这些指标在整个 Autoware 的抽象结构中进行了描述.详细指标不包含在模型中,因为它们会增加模型的复杂性.相反,后续部分将介绍详细指标.与顶级指标相比,详细指标的更新频率更高,这是将它们单独分类的另一个原因.
每个列表都包含一列用于引用值.参考值表示当 Autoware 根据 教程.需要注意的是,reference 值不是必需的值,这意味着 Autoware 不一定在 教程 执行.
Autoware 的顶级时间性能指标#
下图介绍了顶级临时性能指标的模型.
以下三个策略有助于选择顶级性能指标：
- 根据消耗观测值(如传感器数据)的组件拆分 Autoware,并考虑这些组件周围的处理频率和响应时间
- 根据 Planning and Control 的入口点划分 Autoware,并考虑这些组件的处理频率和响应时间
- 显示 Vehicle Interface 的最小指标,因为它们可能因目标车辆而异
此外,还假设算法作为多个节点实现,并用作管道处理系统.
| ID | 模型中的表示 | 指标含义 | 相关功能 | 参考值 | 选择它作为量度的原因 | 注意 |
|---|---|---|---|---|---|---|
| AWOV-001 | Message rate from CPA #9 to CPA #18 | Update rate of result from Prediction to Planning. | Perception | 10 Hz | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | |
| AWOV-002 | Response time from CPA #0 to CPA #20 via CPA #18 | Response time in main body of Perception. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is used if delay compensation is disabled in Tracking. |
| AWOV-003 | Response time from CPA #7 to CPA #20 | Response time from Tracking output of Tracking to its data consumption in Planning. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is used if delay compensation is enabled in Tracking. |
| AWOV-004 | Response time from CPA #0 to CPA #6 | Duration to process pointcloud data in Sensing and Detection. | Perception | N/A | Tracking relies on detection to provide real-time and up-to-date sensed data for accurate tracking. | The metric is used if delay compensation is enabled in Tracking. |
| AWOV-005 | Message rate from CPA #4 to CPA #5 | Update rate of Detection result received by Tracking. | Perception | 10 Hz | Tracking relies on detection to provide real-time and up-to-date sensed data for accurate tracking. | |
| AWOV-006 | Response time from CPA #0 to CPA #14 | Response time from output of observed data from LiDARs to its consumption in EKF Localizer via NDT Scan Matcher. | Localization | N/A | EKF Localizer relies on fresh and up-to-date observed data from sensors for accurate estimation of self pose. | |
| AWOV-007 | Message rate from CPA #11 to CPA #13 | Update rate of pose estimated by NDT Scan Matcher. | Localization | 10 Hz | EKF Localizer relies on fresh and up-to-date observed data from sensors for accurate estimation of self pose. | |
| AWOV-008 | Message rate from CPA #15 to CPA #12 | Update rate of feed backed pose estimated by EKF Localizer. | Localization | 50 Hz | NDT Scan Matcher relies on receiving estimated pose from EKF Localizer smoothly for linear interpolation. | |
| AWOV-009 | Message rate from CPA #17 to CPA #19 | Update rate of Localization result received by Planning. | Localization | 50 Hz | Planning relies on Localization to update the estimated pose frequently. | |
| AWOV-010 | Response time from CPA #20 to CPA #23 | Processing time from beginning of Planning to consumption of Trajectory message in Control. | Planning | N/A | A vehicle relies on Planning to update trajectory within a short time frame to achieve safe driving behavior. | |
| AWOV-011 | Message rate from CPA #21 to CPA #22 | Update rate of Trajectory message from Planning. | Planning | 10 Hz | A vehicle relies on Planning to update trajectory frequently to achieve safe driving behavior. | |
| AWOV-012 | Message rate from CPA #24 to CPA #25 | Update rate of Control command. | Control | 33 Hz | Control stability and comfort relies on sampling frequency of Control. | |
| AWOV-013 | Message rate between CPA #26 and Vehicle | Communication rate between Autoware and Vehicle. | Vehicle Interface | N/A | A vehicle requires Autoware to communicate with each other at predetermined frequency. | Temporal performance requirement varies depending on vehicle type. |
注意
假设每个传感器(例如 LiDAR 和相机)都会输出一组带有时间戳的点云.CPA #0 与时间戳一起观察.如果传感器未配置为输出时间戳,则改用 Autoware 接收点云的时间.这在模型中由 CPA #1 表示.详细指标也采用了这个想法.
Perception 的详细时态性能指标#
下图介绍了 Perception 的临时性能指标模型.
以下两个策略有助于选择性能指标：
- 关于在 Planning 中使用物体识别和红绿灯识别结果的频率和响应时间
- 在合并来自多个处理路径的数据点时拆分 Perception 组件,并考虑该点周围的频率和响应时间
以下列表显示了 Perception 的临时性能指标.
| ID | 模型中的表示 | 指标含义 | 相关功能 | 参考值 | 选择它作为量度的原因 | 注意 |
|---|---|---|---|---|---|---|
| APER-001 | Message rate from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Traffic Light Recognition | 10 Hz | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition for making precise decisions. | |
| APER-002 | Response time from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Traffic Light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition for making precise decisions. | |
| APER-003 | Message rate from CPP #25 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Object Recognition | 10 Hz | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is same as AWOV-001. |
| APER-004 | Response time from CPP #6 to CPP #30 | Response time from Tracking output of Tracking to its data consumption in Planning. | Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is same as AWOV-002 and used if delay compensation is disabled in Tracking. |
| APER-005 | Response time from CPP #23 to CPP #30 | Response time from Tracking output of Tracking to its data consumption in Planning. | Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is same as AWOV-003 and used if delay compensation is enabled in Tracking. |
| APER-006 | Response time from CPP #6 to CPP #21 | Duration to process pointcloud data in Sensing and Detection. | Object Recognition | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | The metrics is same as AWOV-004 and used if delay compensation is enabled in Tracking. |
| APER-007 | Message rate from CPP #20 to CPP #21 | Update rate of Detection result received by Tracking. | Object Recognition | 10 Hz | Tracking relies on detection to provide real-time and up-to-date sensed data for accurate tracking. | The metric is same as AWOV-005 |
| APER-008 | Message rate from CPP #14 to CPP #19 | Update rate of data sent from Sensor Fusion. | Object Recognition | 10 Hz | Association Merger relies on the data to be updated at expected frequency for data synchronization. | |
| APER-009 | Message rate from CPP #16 to CPP #19 | Update rate of data sent from Detection by Tracker. | Object Recognition | 10 Hz | Association Merger relies on the data to be updated at expected frequency for data synchronization. | |
| APER-010 | Message rate from CPP #18 to CPP #19 | Update rate of data sent from Validation | Object Recognition. | 10 Hz | Association Merger relies on the data to be updated at expected frequency for data synchronization. | |
| APER-011 | Response time from CPP #6 to CPP #19 via CPP #14 | Response time to consume data sent from Sensor Fusion after LiDARs output pointcloud. | Object Recognition | N/A | Association Merger relies on fresh and up-to-date data for data synchronization. | |
| APER-012 | Response time from CPP #6 to CPP #19 via CPP #16 | Response time to consume data sent from Detection by Tracker after LiDARs output pointcloud. | Object Recognition | N/A | Association Merger relies on fresh and up-to-date data for data synchronization. | |
| APER-013 | Response time from CPP #6 to CPP #19 via CPP #18 | Response time to consume data sent from Validator after LiDARs output pointcloud. | Object Recognition | N/A | Association Merger relies on fresh and up-to-date data for data synchronization. | |
| APER-014 | Message rate from CPP #10 to CPP #13 | Update rate of data sent from Clustering. | Object Recognition | 10 Hz | Sensor Fusion relies on the data to be updated at expected frequency for data synchronization. | |
| APER-015 | Message rate from CPP #5 to CPP #13 | Update rate of data sent from Camera-based Object detection. | Object Recognition | 10 Hz | Sensor Fusion relies on the data to be updated at expected frequency for data synchronization. | |
| APER-016 | Response time from CPP #6 to CPP #13 | Response time to consume data sent from Clustering after LiDARs output pointcloud. | Object Recognition | N/A | Sensor Fusion relies on fresh and up-to-date data for data synchronization. | |
| APER-017 | Response time from CPP #3 to CPP #13 | Response time to consume data sent from Camera-based Object detection after Cameras output images. | Object Recognition | N/A | Sensor Fusion relies on fresh and up-to-date data for data synchronization. | |
| APER-018 | Message rate from CPP #10 to CPP #17 | Update rate of data sent from Clustering. | Object Recognition | 10 Hz | Validator relies on the data to be updated at expected frequency for data synchronization. | It seems similar to APER-014, but the topic message is different. |
| APER-019 | Message rate from CPP #12 to CPP #17 | Update rate of data sent from DNN-based Object Recognition. | Object Recognition | 10 Hz | Validator relies on the data to be updated at expected frequency for data synchronization. | |
| APER-020 | Response time from CPP #6 to CPP #17 via CPP #10 | Response time to consume data sent from Clustering after LiDARs output pointcloud. | Object Recognition | N/A | Validator relies on fresh and update-date data for data synchronization. | It seems similar to APER-015, but the topic message is different. |
| APER-021 | Response time from CPP #6 to CPP #17 via CPP #12 | Response time to consume data sent from DNN-based Object Recognition after LiDARs output pointcloud. | Object Recognition | N/A | Validator relies on fresh and update-date data for data synchronization. |
Paths between Obstacle segmentation and Planning 的详细时间性能指标#
障碍物分割是 Perception 的关键部分,它将数据传输到 Planning.下图说明了考虑了与 Obstacle segmentation 和 Planning 相关的性能指标的模型.
注意
障碍网格地图和 Obstacle segmentation 都会将数据传输到 Planning 的多个子组件.但是,模型中并未描述所有这些子组件.这是因为我们的主要关注点是通过障碍物分割从 LiDAR 到 Planning 的路径.
以下列表显示了有关 Obstacle segmentation 和 Planning 的时间性能指标.
| ID | 模型中的表示 | 指标含义 | 相关功能 | 参考值 | 选择它作为量度的原因 | 注意 |
|---|---|---|---|---|---|---|
| OSEG-001 | Message rate from CPS #4 to CPS #7 | Update rate of Occupancy grid map received by Planning (behavior_path_planner) |
Obstacle segmentation | 10 Hz | Planning relies on Occupancy grid map to be updated frequently and smoothly for creating accurate trajectory. | |
| OSEG-002 | Response time from CPS #0 to CPS #9 via CPS #7 | Response time to consume Occupancy grid map after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Occupancy grid map for creating accurate trajectory.. | |
| OSEG-003 | Message rate from CPS #6 to CPS #11 | Update rate of obstacle segmentation received by Planning (behavior_velocity_planner). |
Obstacle segmentation | 10 Hz | Planning relies on Obstacle segmentation to be updated frequently and smoothly for creating accurate trajectory. | |
| OSEG-004 | Response time from CPS #0 to CPS #13 via CPS #11 | Response time to consume Obstacle segmentation after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation for creating accurate trajectory.. |