Beyond Visual Perception: Insights from Smartphone Interaction of Visually Impaired Users with Large Multimodal Models (2025)

Procedure

We conducted individual semi-structured interviews via Zoom lasting 50-76 minutes, with one or two researchers present per session. All interviews were recorded with participant consent.

The interviews followed four main phases.First, we invited participants to share their personal use cases for BeMyAI.To facilitate recall, we referenced Be My Eyes’ common use case list(bma, 2024), prompting participants to discuss similar experiences.Follow-up questions were posed to further investigate their experiences with each identified use case.Second, participants evaluated the quality of the tool’s visual interpretations, focusing on accuracy, detail level, error, and appropriateness of identity descriptions.Third, we explored how participants used BMA among their various assistive tools, including human-assisted and AI-powered VQA systems, to understand its distinct advantages and limitations.Finally, we collected demographic information and inquired about participants’ willingness to share copies of its visual interpretations for research analysis.

From Participants

Participants were given the flexibility to select their preferred documentation method, either copying text or taking screenshots.In total, we gathered 22 image descriptions from 4 participants, as most participants either lacked examples or were reluctant to share due to privacy concerns or the effort required in documentation.Of these descriptions, 4 were original images alongside their descriptions, while the majority consisted of text copies – a format participants found more manageable.Additionally, 6 of the 22 descriptions contained follow-up questions where participants sought clarification or additional details from the system.

From Social Media Platforms

We collected image descriptions from 4 social media platforms: X¹¹1http://x.com/, Facebook²²2https://www.facebook.com/, Reddit³³3https://www.reddit.com/, and blogposts. The platforms were chosen to cover both image-sharing sites (X, Facebook) and discussion forums (Reddit, blogposts).

We used the search terms “BeMyAI” and “#bemyai” to find relevant posts on X, Facebook, and blogposts. For Reddit, we searched for “AI” and “BeMyAI” within the r/Blind community.

We gathered posts that met the following criteria: (i) written in English, (ii) published between September 25, 2023, the official release date of BeMyAI, and March 31, 2024, and (iii) contained image descriptions generated by the tool.To avoid redundancy, if identical content appeared across multiple platforms, only the earliest published post was recorded.If one post included multiple descriptions, each description was documented as a separate entry.If there were multiple posts published around the same time frame about the same topic (e.g., an X or Reddit thread), they were treated as a single entry.

We verified the authenticity of posts by confirming that they were from either the official Be My Eyes account or from users who identified themselves as visually impaired in their profiles. These established platforms serve as reliable sources where PVI regularly share their experiences.

In total, we collected 28 BMA-generated image descriptions from 4 social media platforms. Of these, 23 included original images sent to BMA and 5 were text-only interactions. Fifteen descriptions contained follow-up questions that participants sent to the system for additional details or clarification.By platform, we collected(i) X: 17 image descriptions (16 with original images, 6 with follow-up questions),(ii) Facebook: 3 image descriptions (all with original images),(iii) Reddit: 3 image descriptions (1 with follow-up questions), and(iv) blogposts: 5 image descriptions (4 with original images, 8 with follow-up questions).The complete dataset is available at https://bit.ly/4hqo5ve.

This dataset, although limited in size, captures crucial insights into early user experiences with BeMyAI. By collecting descriptions shortly after its release, we documented initial user interactions and system performance during its early phase.

Capabilities of BeMyAI

Seven participants (P2, P3, P7, P9, P11, P13, P14) reported that the AI tool’s comprehensive scene descriptions enhanced their spatial awareness.These descriptions often uncover spatial details previously unnoticed by users, as noted by P11, “It gives me unexpected information about things that I didn’t even know were there.”

However, this enhanced awareness is compromised by AI hallucinations, instances where AI incorrectly identifies nonexistent objects.Such errors can disrupt the user’s context understanding, leading to confusion or misinterpretation of the environment. To verify BMA’s descriptions, participants either seek human assistance or rely on their pre-existing mental models of the environment.

Participants successfully guide BMA using auditory cues, particularly when locating dropped objects. They combine their hearing and spatial memory to adjust camera angles, which enables the tool to scan more accurately.This integration of participants’ auditory inputs guides and refines the system performance, leading to more accurate and useful descriptions.

Example 1: AI Hallucinations of Adding Non-existent Details

Be My AI provides detailed descriptions of objects, characterizing their colors, sizes (e.g., “small,” “medium,” “large,” and “tall”), shapes (e.g., “round,” “square,” “fluffy,” “wispy,” “dense,” and “open lattice structure”), and spatial orientations (e.g., “on the left,” “to the right,” “in the foreground,” “in the middle,” and “in the background”).However, four participants (P3, P5, P11, P14) encountered AI hallucinations where the tool fabricated details in their environments.

Participants developed two strategies to verify the accuracy: consulting human assistants and drawing on their spatial memory.P3’s experience illustrates the value of human verification. When BeMyAI reported a “phantom” object behind her, she first investigated personally, finding nothing. A human assistant then confirmed the absence of any object.

P5’s experience demonstrates how spatial memory helps identify inaccuracies. When the system incorrectly described objects next to her dogs in her home, her familiarity with the space enabled her to recognize these errors independently. This ability to leverage environmental knowledge allows participants to detect and disregard AI hallucinations effectively.

Example 2: Need for User Support in Locating Dropped Objects

Participants (P4, P12, P13) used BMA’s sequential “top to bottom, left to right” descriptions to locate dropped items like earbuds and hair ties.These descriptions provided precise locations, such as when P13 learned of “a picture of a carpet with a hair tie in the upper right hand corner,” and when P12 discovered “the earphones are directly in front of you, between your feet.”Such sequential descriptions allow participants to pinpoint lost objects with greater accuracy.

To optimize the tool’s scanning effectiveness, participants first used their auditory perception and spatial memory to approximate an object’s location before positioning the camera.This adjustment allowed the tool to focus on the intended search area, rather than random searching.For instance, P4 utilized his “listening skills” to detect the location of a fallen object before directing the camera to that specific spot. Similarly, P12 enhanced the camera’s view of dropped earbuds by stepping back from his seat to capture a better angle of the floor.These examples illustrate how participants synergize their understanding of the environment with the system’s technological capabilities to manage tasks that require spatial awareness.

Capabilities of BeMyAI

BMA usually concludes its descriptions by injecting subjective interpretations, which enriches users’ understanding of depicted scenarios.These interpretations encompass people’s or animal’s emotional states (e.g., “a cheerful expression on his face,” “smiling slightly,” “friendly and approachable,” “curious expression”),body language (e.g., “his arms are open as if he is engaging in a conversation or greeting the other man”),and the ambiance (e.g., “peaceful and natural,” “serene and peaceful atmosphere,” “cozy and cheerful holiday vibe,” “warm and inviting atmosphere,” “the atmosphere seems to be lively and festive”).

These subjective interpretations help participants better engage in social interactions and understand animal behavior.Participants (P5, P9-11) specifically used the system to grasp nuances in animals’ facial expressions, activities, and body language (Figure1). P10 highlighted its utility during dog walks: “Sometimes it’s hard for me to know if the dog is peeing or what the dog is doing.”

Furthermore, the subjectivity extends to fashion suggestions, with BeMyAI recommending stylistically coordinated outfits based on colors and patterns, as well as assessing makeup.For instance, P5 employed it to check the color, placement, and overall balance of her makeup, P2 and P9 consulted it to coordinate tops and bottoms, and P12 used it to select a tie that complemented his shirt.

However, participants noted that BMA’s visual interpretations sometimes included excessive subjective elements.This subjective input, unverified by human, could potentially undermine the system’s ability to provide accurate context awareness.As a result, participants preferred their own subjective interpretations or feedback from sighted assistants in contexts involving human-animal interactions and fashion choices.

Example 1: Subjective Interpretations in Human-Animal Interactions

Participants (P2, P5) identified limitations in how BMA infers animal emotions in its image descriptions, raising concerns about the accuracy of these subjective interpretations.For instance, P5 highlighted instances where the tool describes a dog “appears to be relaxed” or “appears to be happy.”Likewise, P2 noted its tendency to include subjective commentary, as in “That’s a white cat curled up on a fuzzy blanket. She looks peaceful and happy and rested.”

These interpretations go beyond observable visual elements to make emotional inferences that may not reflect reality.As a result, participants expressed a preference for objective, fact-based descriptions.P2 articulated a desire for less editorializing, saying, “Maybe I don’t want it to editorialize, you know, maybe I just literally only want the facts of it.”

Participants (P2, P5) stressed the importance of maintaining human agency in interpreting animal behaviors, preferring their own judgment rather than the system’s subjective interpretations.P5 particularly valued the ability to modify BMA’s descriptions of her dog’s expressions, maintaining control over her interpretation of pet behaviors.

In summary, BMA’s subjective inferences can enrich descriptions, yet they risk introducing inaccuracies that undermine the system’s reliability. Users therefore value the ability to override these interpretations, preserving their agency in understanding the context.

Example 2: Subjective Interpretations in Fashion Help

BMA’s subjective interpretations extend beyond human-animal interactions to fashion help.Participants (P6, P7, P10) utilized it to describe colors and patterns but expressed concerns about its subjective fashion suggestions.They preferred to make their own style choices or seek human assistance for outfit matching, highlighting the human subjectivity in fashion decisions.

P6 questioned the AI’s capacity to authentically replicate human judgment, saying, “It’s interesting how AI is being taught to simulate kind of the human factor of things.”She cited experiences where AI-generated responses appeared “strange” and “complete nonsense,” contrasting these with the nuanced understanding that humans provide.

P10 explicitly preferred human feedback, stating, “I’m still more confident asking a sighted person to provide me with the feedback.”This preference underscores how participants maintain their agency by relying on human judgment for fashion decisions rather than deferring to BMA’s suggestions.

Capabilities of BeMyAI

BMA describes people’s identity in terms of gender (e.g., “woman,” “man”), age (e.g., “old,” “late twenties or early thirties”), appearance (e.g., “long, dark brown hair,” “wavy brown hair,” “His hair falls past his ears, with a slightly messy but stylish look”) and ethnicity (e.g., “East Asian”).Participants appreciated such detailed descriptions for providing deeper insight into people’s visual characteristics. P2 explained: “I enjoyed hearing, I knew my friend was East Asian when I saw her picture. It was kind of cool to see that because… as a blind person, you don’t know that all the time. So, I’d like access to it.”

However, eight participants (P2, P3, P5-7, P9, P10, P13) encountered challenges with the system’s accuracy and sensitivity in describing identity attributes, particularly gender and age.These errors reveal the tool’s limited ability to interpret contextual cues that extend beyond visual appearance, such as cultural, situational, and personal contexts.

Example: Inaccurate Identification of People’s Gender and Age

Three participants (P3, P10, P13) reported inaccuracies in BMA’s gender identification.P13 described how it misidentified gender by mistaking a hidden ponytail for short hair: “just looked like short-cropped hair.”Similarly, P10 revealed that the system struggled with gender identification for individuals whose physical attributes do not conform to typical gender norms, noting, “the person had a short hair, and was a female”.In response, it defaulted to neutral language, “BeMyAI didn’t tell me it was a woman or a man, just said a person.”These examples illustrate the system’s reliance on stereotypical indicators and reveal its challenges in interpreting non-visible details.

Additionally, two participants (P5, P9) identified problems with age description. P9 reported that BeMyAI overestimated her daughter’s age by one year.Likewise, P5 observed sensitivity concerns when the system labeled someone as “an old woman” based solely on grey hair.To address these age-related inaccuracies, P5 suggested that the tool should adopt more objective, factual descriptions rather than subjective or potentially stigmatizing labels, such as “a woman with grey hair, instead of an older woman.”These examples indicate the system’s difficulties with precise age estimation and the potential inaccuracy when AI makes assumptions based on appearance alone.

In summary, these examples illustrate the tool’s limited context awareness and its challenge in accurately interpreting people’s identities, such as gender and age, due to the reliance on stereotypical visual cues. These misjudgements are rooted in the system’s inability to integrate and interpret broader, non-visible contextual elements like cultural norms and personal styling choices.

Capabilities of BeMyAI

BMA’s “ask more” function enables users to explore image details that were not covered in initial descriptions. Through this feature, users can probe for specific details that match their personal needs or interests.Eleven participants (P1-3, P5, P6, P8-12, P14) used the “ask more” function tomatch outfits (P2, P5, P12),check makeup (P5, P10),suggest cooking recipes (P12),assist with household appliances (P5, P12),examine text or objects (P6, P8, P11, P12, P14),gain more details about people’s facial expressions, attire, or actions (P2, P3, P5, P8, P9, P10, P11),check animal status (P3, P5, P11, P14),and edit the descriptions for social media posts (P2, P3).

Seven participants (P1-3, P5, P8-10) valued the flexibility of posing follow-up questions, enhancing their independence by reducing reliance on human assistance. As P3 noted, “I can ask follow-up questions, so I have a good way to sort of figure out what’s in the image independently, which is something I was not able to do before this app came out.”

However, participants identified a common issue that the system is unable to discern users’ goals in initial response, often resulting in generalized descriptions without knowing which aspects to emphasize. P12 elaborated on this challenge, stating, “BeMyAI loves to make general descriptions, and it doesn’t know what to focus on.”To overcome this limitation, users guide the system with targeted prompts that clarify their specific intentions.

Example 1: Check for the Presence of Eggshells

In Figure2, BMA’s initial response provides a detailed descriptions of what is included in a frying pan.The user, likely influenced by prior experiences of inadvertently leaving unwanted items in the pan while cooking, specifically intended to check for such elements.However, the system did not infer that the user was looking for unwanted elements like eggshells, rather than seeking a general description of the scene.To clarify her objective, the user posed a follow-up question, thus directing BMA to recognize and prioritize her specific concerns and intentions.

In this example, although the AI tool can describe the visual elements accurately and comprehensively (from the condition of the eggs to the surroundings), it failed to identify unusual visual elements that are challenging for PVI to detect but are important to their understanding of the scene.User prompting helps refocus the system’s attention on these unusual elements, thereby enhancing its ability to interpret and respond to user-specific intents more effectively.

Example 2: Adjust Rotary Control Appliances

Six participants highlighted the tool’s inadequacy in comprehending their goal of adjusting rotary control appliances like washers (P1) and thermostats (P2, P4, P5, P13, P14). This task requires BeMyAI to interpret the current settings on these appliances as users modify dials for time, mode, or temperature.However, the system often offered broad descriptions of visual elements without honing in on the user’s specific objectives.

P13’s experience exemplifies this limitation. While the system could recognize a thermostat on the wall, it failed to provide critical information such as the current temperature setting or instructions for adjusting the temperature.

To overcome the limitation, participants guided the system to better understand their goal through prompting.For instance, P5 asked precise questions like “What is the arrow pointed at right now on the current setting?” Such specific queries helped redirect BMA’s focus from general scene descriptions to the exact details needed for appliance adjustment.

Capabilities of BeMyAI

BMA often fails to proactively suggest next steps or support users until they achieve their objectives.Effective consistency requires maintaining focus on the user’s objective throughout the interaction. This includes offering progressively specific and relevant assistance, and ensuring all responses contribute to the user’s intended outcome.

Due to these limitations, participants turn to human assistants. Human assistants can interpret the context of tasks and goals more dynamically, providing conversational guidance and adapting to users’ actions to facilitate goal completion.

Example 1: Inadequacy in Identifying Central Puzzle Piece

P14 used BMA to differentiate between various puzzles by describing the images on the boxes like a scene of cats or bears.When tasked with identifying the centerpiece of a nine-piece puzzle using the box image, BMA failed to provide the necessary detail for this precise task.Initially, the system correctly read text from the puzzle box indicating which piece belonged in the center.However, when P14 requested more specific guidance to locate the centerpiece, BMA only noted that it was square – a characteristic shared by multiple pieces. This response was too vague for successful puzzle assembly.

Example 2: Lack of Instruction for Camera Adjustment

A common challenge for participants (P3, P5, P6, P8, P10-12, P14) was aligning the camera properly to capture clear views of their intended areas.BeMyAI can identify image quality issues, notifying users when pictures were “cut off” (P5), “blurry” (P6), or incomplete (Figure3).However, it is unable to provide further actionable guidance on adjusting the camera to improve image clarity.This limitation manifested in scenarios where BMA recognized both the user’s goal of capturing specific areas and the image deficiencies but could not suggest practical solutions.As P6 described, “It was very frustrating because it said the picture is blurry. Can you please put the label in the frame? But I didn’t know how to put the label on the frame.”

Eight participants (P1-3, P7, P8, P12-14) resolved this challenge by turning to human assistants, who can offer adaptive support to achieve their goals, especially in tasks that require continuous feedback. P13 shared an example where a human assistant not only understood her goal but also guided her in adjusting the camera and instantly reminded her to turn on the light to achieve her objective.

This example highlights human assistants’ ability to adapt their communication based on the situation and the user’s implied needs, providing solutions that directly support achieving the user’s goals.However, BeMyAI struggles to provide practical assistance despite understanding basic requests.

Capabilities of BeMyAI

Participants (P2, P5, P10) employed BMA to aid in localization and orientation while navigating to their destinations.For instance, P2 utilized it to identify gate numbers at the airport, P5 employed it to read signage directing toward the airport’s transportation area, and P10 used it to recognize her surroundings when disoriented in her neighborhood.Despite these benefits, participants encountered challenges when using BMA for navigation. The primary limitations stemmed from the limited camera view and practical mobility issues.

Example 1: Limited Navigational Information in Static Images

Six participants (P2, P5, P6, P10, P11, P13) reported that BMA’s reliance on static images rather than real-time videos makes it difficult to capture comprehensive navigational information, such as obstacles and signage, in a single shot.As P6 described, users “have to stand there and keep taking pictures and taking pictures,” verify the captured content, assess its utility for navigation, and adjust the angle for additional shots. This iterative process can be time-consuming, “It’ll be too much of a task that’s supposed to take maybe 10 minutes would probably take like 30.”

P10 elaborated on the challenge and risk of simultaneously taking pictures and navigating, particularly when attempting to identify and navigate around obstacles. The uncertainty of capturing all potential hazards was a significant concern.

Given these limitations, participants preferred human assistance through video-based interactions over BeMyAI for real-time navigation.Video interactions not only benefit from human adaptability in adjusting guidance (Section4.2.2) but also offer crucial real-time feedback unavailable in static images.Participants (P2, P6, P13) suggested that real-time video interpretation capabilities would significantly enhance the system by eliminating the need for repeated picture-taking.

Example 2: Irreplaceable Role of Orientation & Mobility Skills in Navigation

Besides the value of real-time visual interpretations from external resources like human assistants, participants (P5, P10, P13, P14) emphasized the indispensable role of real-time feedback through their Orientation and Mobility (O&M) skills for safe navigation.P5 and P13 pointed out that even human assistance, though adaptive in guiding PVI away from navigational hazards, cannot substitute for essential O&M skills required for tasks like street crossing.Consequently, participants are more cautious about relying on emerging AI tools for navigation.P14 reinforced this perspective, saying, “it’s not a replacement for our mobility skills or just any skills in general. It can aid and augment the skills but it’s not a replacement for them. It shouldn’t be.”

Participants (P5, P13) further delineated vital information provided by O&M skills or discerned through O&M tools like white canes or guide dogs, which current AI or human assistance cannot replace.First, immediate surroundings. BeMyAI can identify obstacles “that are a little far away” (P5) but may miss immediate surrounding hazards.Second, distance and proximity measurements of obstacles. Although BMA can indicate the presence of obstacles, it lacks the capability to measure their distance.Third, directional details. Essential navigational information, such as the direction of stairs (ascending or descending) or the presence of railings, are not always detectable through the system.With the O&M skills and tools, users can instantly adapt their movements based on direct interaction with their environment. This level of responsiveness is currently unattainable with AI tools or human assistance.

In summary, neither AI systems nor human assistants can replace the essential, real-time feedback and adaptive capabilities offered by O&M skills and tools. These elements are vital for ensuring safe navigation by allowing users to directly interact with their environment.