publications

2025

1. 

   Sightation Counts: Leveraging Sighted User Feedback in Building a BLV-aligned Dataset of Diagram Descriptions

   Wan Ju Kang, Eunki Kim, Na Min An, and 4 more authors

   arXiv preprint arXiv:2503.13369, 2025

   Abs arXiv

   Often, the needs and visual abilities differ between the annotator group and the end user group. Generating detailed diagram descriptions for blind and low-vision (BLV) users is one such challenging domain. Sighted annotators could describe visuals with ease, but existing studies have shown that direct generations by them are costly, bias-prone, and somewhat lacking by BLV standards. In this study, we ask sighted individuals to assess – rather than produce – diagram descriptions generated by vision-language models (VLM) that have been guided with latent supervision via a multi-pass inference. The sighted assessments prove effective and useful to professional educators who are themselves BLV and teach visually impaired learners. We release Sightation, a collection of diagram description datasets spanning 5k diagrams and 137k samples for completion, preference, retrieval, question answering, and reasoning training purposes and demonstrate their fine-tuning potential in various downstream tasks.

2. 

   Can LVLMs and Automatic Metrics Capture Underlying Preferences of Blind and Low-Vision Individuals for Navigational Aid?

   Na Min An*, Eunki Kim*, Wan Ju Kang, and 3 more authors

   arXiv preprint arXiv:2502.14883, 2025

   * indicates equal contribution.

   Abs arXiv

   Vision is a primary means of how humans perceive the environment, but Blind and Low-Vision (BLV) people need assistance understanding their surroundings, especially in unfamiliar environments. The emergence of semantic-based systems as assistance tools for BLV users has motivated many researchers to explore responses from Large Vision-Language Models (LVLMs). However, it has yet been studied preferences of BLV users on diverse types/styles of responses from LVLMs, specifically for navigational aid. To fill this gap, we first construct Eye4B dataset, consisting of human-validated 1.1k curated outdoor/indoor scenes with 5-10 relevant requests per scene. Then, we conduct an in-depth user study with eight BLV users to evaluate their preferences on six LVLMs from five perspectives: Afraidness, Nonactionability, Sufficiency, and Conciseness. Finally, we introduce Eye4B benchmark for evaluating alignment between widely used model-based image-text metrics and our collected BLV preferences. Our work can be set as a guideline for developing BLV-aware LVLMs towards a Barrier-Free AI system.

3. 

   Image Embedding Sampling Method for Diverse Captioning

   Sania Waheed*, and Na Min An*

   arXiv preprint arXiv:2502.10118, 2025

   * indicates equal contribution.

   Abs arXiv

   Image Captioning for state-of-the-art VLMs has significantly improved over time; however, this comes at the cost of increased computational complexity, making them less accessible for resource-constrained applications such as mobile devices and assistive technologies. Alternatively, smaller VLMs prioritize high-level scene descriptions, overlooking finer details that contribute to a richer understanding of an image. In this paper, we introduce a training-free framework that enhances caption diversity and informativeness by explicitly attending to distinct image regions using a comparably small VLM, BLIP, as the backbone. Our approach leverages structured segmentation to produce hierarchical representations that capture both global and localized semantics. Without requiring additional model training, we demonstrate that our method allows smaller VLMs to achieve performance comparable to larger models in terms of image-caption alignment, semantic integrity, and diversity. We evaluate our framework on MSCOCO, Flickr30k, and Nocaps test datasets, achieving a Div-2 score of 0.735, 0.750, and 0.748 for each dataset respectively, while maintaining strong image-caption relevancy and semantic integrity with the human-annotated captions.

4. 

   When Tom Eats Kimchi: Evaluating Cultural Bias of Multimodal Large Language Models in Cultural Mixture Contexts

   Jun Seong Kim*, Kyaw Ye Thu*, Javad Ismayilzada, and 6 more authors

   Accepted to Proceedings of the 3rd Workshop on Cross-Cultural Considerations in NLP (NAACL), 2025

   * indicates equal contribution.

   Abs arXiv

   In a highly globalized world, it is important for multi-modal large language models (MLLMs) to recognize and respond correctly to mixedcultural inputs. For example, a model should correctly identify kimchi (Korean food) in an image both when an Asian woman is eating it, as well as an African man is eating it. However, current MLLMs show an over-reliance on the visual features of the person, leading to misclassification of the entities. To examine the robustness of MLLMs to different ethnicity, we introduce MIXCUBE, a cross-cultural bias benchmark, and study elements from five countries and four ethnicities. Our findings reveal that MLLMs achieve both higher accuracy and lower sensitivity to such perturbation for high-resource cultures, but not for low-resource cultures. GPT-4o, the best-performing model overall, shows up to 58% difference in accuracy between the original and perturbed cultural settings in low-resource cultures.

5. 

   Diffusion Models Through a Global Lens: Are They Culturally Inclusive?

   Zahra Bayramli*, Ayhan Suleymanzade*, Na Min An, and 5 more authors

   Accepted to Proceedings of the 3rd Workshop on Cross-Cultural Considerations in NLP (NAACL), 2025

   * indicates equal contribution.

   Abs arXiv

   Text-to-image diffusion models have recently enabled the creation of visually compelling, detailed images from textual prompts. However, their ability to accurately represent various cultural nuances remains an open question. In our work, we introduce CultDiff benchmark, evaluating state-of-the-art diffusion models whether they can generate culturally specific images spanning ten countries. We show that these models often fail to generate cultural artifacts in architecture, clothing, and food, especially for underrepresented country regions, by conducting a fine-grained analysis of different similarity aspects, revealing significant disparities in cultural relevance, description fidelity, and realism compared to real-world reference images. With the collected human evaluations, we develop a neural-based image-image similarity metric, namely, CultDiff-S, to predict human judgment on real and generated images with cultural artifacts. Our work highlights the need for more inclusive generative AI systems and equitable dataset representation over a wide range of cultures.

6. 

   Machine Learning Techniques for Simulating Human Psychophysical Testing of Low-Resolution Phosphene Face Images in Artificial Vision

   Na Min An, Hyeonhee Roh, Sein Kim, and 2 more authors

   Advanced Science, 2025

   Abs PDF

   To evaluate the quality of artificial visual percepts generated by emerging methodologies, researchers often rely on labor-intensive and tedious human psychophysical experiments. These experiments necessitate repeated iterations upon any major/minor modifications in the hardware/software configurations. Here, the capacity of standard machine learning (ML) models is investigated to accurately replicate quaternary match-to-sample tasks using low-resolution facial images represented by arrays of phosphenes as input stimuli. Initially, the performance of the ML models trained to approximate innate human facial recognition abilities across a dataset comprising 3600 phosphene images of human faces is analyzed. Subsequently, due to the time constraints and the potential for subject fatigue, the psychophysical test is limited to presenting only 720 low-resolution phosphene images to 36 human subjects. Notably, the superior model adeptly mirrors the behavioral trend of human subjects, offering precise predictions for 8 out of 9 phosphene quality levels on the overlapping test queries. Subsequently, human recognition performances for untested phosphene images are predicted, streamlining the process and minimizing the need for additional psychophysical tests. The findings underscore the transformative potential of ML in reshaping the research paradigm of visual prosthetics, facilitating the expedited advancement of prostheses.

2024

1. 

   I0T: Embedding Standardization Method Towards Zero Modality Gap

   Na Min An*, Eunki Kim*, James Thorne, and 1 more author

   arXiv preprint arXiv:2412.14384, 2024

   * indicates equal contribution.

   Abs arXiv

   Contrastive Language-Image Pretraining (CLIP) enables zero-shot inference in downstream tasks such as image-text retrieval and classification. However, recent works extending CLIP suffer from the issue of modality gap, which arises when the image and text embeddings are projected to disparate manifolds, deviating from the intended objective of image-text contrastive learning. We discover that this phenomenon is linked to the modality-specific characteristic that each image/text encoder independently possesses and propose two methods to address the modality gap: (1) a post-hoc embedding standardization method, I0Tpost that reduces the modality gap approximately to zero and (2) a trainable method, I0Tasync, to alleviate the modality gap problem by adding two normalization layers for each encoder. Our I0T framework can significantly reduce the modality gap while preserving the original embedding representations of trained models with their locked parameters. In practice, I0Tpost can serve as an alternative explainable automatic evaluation metric of widely used CLIPScore (CLIP-S).

2. 

   Stable Language Model Pre-training by Reducing Embedding Variability

   Woojin Chung, Jiwoo Hong, Na Min An, and 2 more authors

   In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024

   Abs arXiv PDF

   Stable pre-training is essential for achieving better-performing language models. However, tracking pre-training stability by calculating gradient variance at every step is impractical due to the significant computational costs. We explore Token Embedding Variability (TEV) as a simple and efficient proxy for assessing pre-training stability in language models with pre-layer normalization, given that shallower layers are more prone to gradient explosion (section 2.2). Moreover, we propose Multi-head Low-Rank Attention (MLRA) as an architecture to alleviate such instability by limiting the exponential growth of output embedding variance, thereby preventing the gradient explosion (section 3.2). Empirical results on GPT-2 with MLRA demonstrate increased stability and lower perplexity, particularly in deeper models.

3. 

   Capturing the Relationship Between Sentence Triplets for LLM and Human-Generated Texts to Enhance Sentence Embeddings

   Na Min An, Sania Waheed, and James Thorne

   In Findings of the Association for Computational Linguistics: EACL 2024, Mar 2024

   Abs PDF

   Deriving meaningful sentence embeddings is crucial in capturing the semantic relationship between texts. Recent advances in building sentence embedding models have centered on replacing traditional human-generated text datasets with those generated by LLMs. However, the properties of these widely used LLM-generated texts remain largely unexplored. Here, we evaluate the quality of the LLM-generated texts from four perspectives (Positive Text Repetition, Length Difference Penalty, Positive Score Compactness, and Negative Text Implausibility) and find that there exists an inherent difference between human and LLM-generated datasets. To further enhance sentence embeddings using both human and LLM-generated datasets, we propose a novel loss function that incorporates Positive-Negative sample Augmentation (PNA) within the contrastive learning objective. Our results demonstrate that PNA effectively mitigates the sentence anisotropy problem in Wikipedia corpus (-7% compared to CLHAIF) and simultaneously improves the Spearman’s correlation in standard Semantic Textual Similarity (STS) tasks (+1.47% compared to CLHAIF).

2023

1. 

   Can Large Language Models Capture Dissenting Human Voices?

   Noah Lee*, Na Min An*, and James Thorne

   In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), Dec 2023

   * indicates equal contribution.

   Abs arXiv PDF

   Large language models (LLMs) have shown impressive achievements in solving a broad range of tasks. Augmented by instruction fine-tuning, LLMs have also been shown to generalize in zero-shot settings as well. However, whether LLMs closely align with the human disagreement distribution has not been well-studied, especially within the scope of natural language inference (NLI). In this paper, we evaluate the performance and alignment of LLM distribution with humans using two different techniques to estimate the multinomial distribution: Monte Carlo Estimation (MCE) and Log Probability Estimation (LPE). As a result, we show LLMs exhibit limited ability in solving NLI tasks and simultaneously fail to capture human disagreement distribution. The inference and human alignment performances plunge even further on data samples with high human disagreement levels, raising concerns about their natural language understanding (NLU) ability and their representativeness to a larger human population.

2. 

   Reinforcement Learning Framework to Simulate Short-Term Learning Effects of Human Psychophysical Experiments Assessing the Quality of Artificial Vision

   Na Min An, Hyeonhee Roh, Sein Kim, and 2 more authors

   In 2023 International Joint Conference on Neural Networks (IJCNN), Dec 2023

   Abs PDF

   The quality of the artificial vision produced by visual prostheses has traditionally been evaluated by human psychophysical tests using images expressed with an array of phosphenes. However, such experiments involving human subjects are considerably time-consuming and labor-intensive. One potential solution may be implementing an efficient approach to assist or replace psychophysical experiments showing a short-term learning effect in human subjects. The present work developed a reinforcement learning (RL)-based feedback framework which built artificial agents to emulate the behavioral changes in the learning process of human subjects over the experimental trials. In our framework, we first trained an agent which can gradually learn to identify 720 faces presented in low-resolution phosphene images with feedback rewards received from the agreement in the perception of nine training human subjects. Then, in the automating stage of the framework, we created nine RL agents. By testing those agents, we found the RL agents mimicked the learning effects of nine test human subjects better than nine instances of a supervised learning (SL) model. Given the similar outcomes with human tests and the time efficiency of RL, our framework may expedite the development of visual prosthetic systems by at least partially replacing laborious human psychophysical experiments.

3. 

   Artificial Vision Parameter Learning and Automating Method for Improving Visual Prosthetic Systems

   Maesoon Im, Hyeonhee Roh, Na Min An, and 1 more author

   Jun 2023

   US Patent App. 18/075,555

   PDF

2021

1. 

   Machine Learning Approaches as An Alternative to Human Psychophysical Tests of Prosthetic Vision (abstract)

   Na Min An, Hyeonhee Roh, Soomin Jung, and 2 more authors

   2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Jun 2021

   Abs PDF

   Microelectronic retinal prostheses can evoke artificial visual percepts to blind individuals. Among several factors, expression of original images using limited resources (i.e., small number of pixels and low gray scales) is critical for improved recognition of artificial vision. To assess effectiveness of new image processing techniques, prosthetic researchers have performed psychophysical tests with normally-sighted subjects. Here, we evaluated facial recognition task performances of two machine learning (ML) models for phosphene images in various conditions, representing several electrical stimulation cases. Both ML models showed correct response ratios with trends expected from psychophysical studies with human subjects. Our results suggest ML approaches would be useful for evaluating newly-developed artificial vision image processing techniques. In particular, the use of ML approaches is expected to substantially reduce time and cost needed for psychophysical testing of phosphene images.