학술

MAP: A Benchmark on Multimodal Accessibility Planning for Real World Places

arXiv CS.AI
CC BY
이 매체는 공공·자유 라이선스로 본문을 직접 표시합니다.

Abstract

We introduce MAP, the first benchmark to evaluate multimodal AI systems as assistants for users with accessibility requirements when planning visits to places in the real world.

In our evaluation, systems are presented with requests to verify or recommend a point of interest meeting an accessibility requirement.

MAP contains two novel assessments: Claim verification for accessibility planning assesses if information on places and stated accessibility features is supported and identifies places that satisfy requested accessibility features.

Visual evidence retrieval for accessibility planning checks if a multimodal AI system can select visual evidence for the requested place and accessibility feature.

Our methodology supports comparison of AI systems in a setting where place information and accessibility information can change over time by evaluating systems and refreshing ground truth data at scheduled times.

The benchmark is based on automatic rating and human rating for a proportion of responses.

관련 뉴스