publications

2024

1. 

   Stable Language Model Pre-training by Reducing Embedding Variability

   Woojin Chung, Jiwoo Hong, Na Min An, and 2 more authors

   2024

   PDF

2. 

   Capturing the Relationship Between Sentence Triplets for LLM and Human-Generated Texts to Enhance Sentence Embeddings

   Na Min An, Sania Waheed, and James Thorne

   In Findings of the Association for Computational Linguistics: EACL 2024, Mar 2024

   Abs PDF

   Deriving meaningful sentence embeddings is crucial in capturing the semantic relationship between texts. Recent advances in building sentence embedding models have centered on replacing traditional human-generated text datasets with those generated by LLMs. However, the properties of these widely used LLM-generated texts remain largely unexplored. Here, we evaluate the quality of the LLM-generated texts from four perspectives (Positive Text Repetition, Length Difference Penalty, Positive Score Compactness, and Negative Text Implausibility) and find that there exists an inherent difference between human and LLM-generated datasets. To further enhance sentence embeddings using both human and LLM-generated datasets, we propose a novel loss function that incorporates Positive-Negative sample Augmentation (PNA) within the contrastive learning objective. Our results demonstrate that PNA effectively mitigates the sentence anisotropy problem in Wikipedia corpus (-7% compared to CLHAIF) and simultaneously improves the Spearman’s correlation in standard Semantic Textual Similarity (STS) tasks (+1.47% compared to CLHAIF).

2023

1. 

   Can Large Language Models Capture Dissenting Human Voices?

   Noah Lee*, Na Min An*, and James Thorne

   In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), Dec 2023

   * indicates equal contribution.

   Abs arXiv PDF

   Large language models (LLMs) have shown impressive achievements in solving a broad range of tasks. Augmented by instruction fine-tuning, LLMs have also been shown to generalize in zero-shot settings as well. However, whether LLMs closely align with the human disagreement distribution has not been well-studied, especially within the scope of natural language inference (NLI). In this paper, we evaluate the performance and alignment of LLM distribution with humans using two different techniques to estimate the multinomial distribution: Monte Carlo Estimation (MCE) and Log Probability Estimation (LPE). As a result, we show LLMs exhibit limited ability in solving NLI tasks and simultaneously fail to capture human disagreement distribution. The inference and human alignment performances plunge even further on data samples with high human disagreement levels, raising concerns about their natural language understanding (NLU) ability and their representativeness to a larger human population.

2. 

   Reinforcement Learning Framework to Simulate Short-Term Learning Effects of Human Psychophysical Experiments Assessing the Quality of Artificial Vision

   Na Min An, Hyeonhee Roh, Sein Kim, and 2 more authors

   In 2023 International Joint Conference on Neural Networks (IJCNN), Dec 2023

   Abs PDF

   The quality of the artificial vision produced by visual prostheses has traditionally been evaluated by human psychophysical tests using images expressed with an array of phosphenes. However, such experiments involving human subjects are considerably time-consuming and labor-intensive. One potential solution may be implementing an efficient approach to assist or replace psychophysical experiments showing a short-term learning effect in human subjects. The present work developed a reinforcement learning (RL)-based feedback framework which built artificial agents to emulate the behavioral changes in the learning process of human subjects over the experimental trials. In our framework, we first trained an agent which can gradually learn to identify 720 faces presented in low-resolution phosphene images with feedback rewards received from the agreement in the perception of nine training human subjects. Then, in the automating stage of the framework, we created nine RL agents. By testing those agents, we found the RL agents mimicked the learning effects of nine test human subjects better than nine instances of a supervised learning (SL) model. Given the similar outcomes with human tests and the time efficiency of RL, our framework may expedite the development of visual prosthetic systems by at least partially replacing laborious human psychophysical experiments.

3. 

   Artificial Vision Parameter Learning and Automating Method for Improving Visual Prosthetic Systems

   Maesoon Im, Hyeonhee Roh, Na Min An, and 1 more author

   Jun 2023

   US Patent App. 18/075,555

   PDF

2021

1. 

   Machine Learning Approaches as An Alternative to Human Psychophysical Tests of Prosthetic Vision (abstract)

   Na Min An, Hyeonhee Roh, Soomin Jung, and 2 more authors

   2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), Jun 2021

   PDF