Self-Foveate: Enhancing Diversity and Difficulty of Synthesized Instructions

TL;DR

An LLM-driven Multi-Level Foveation method
for synthesizing high-quality instruction data from unsupervised text
with enhanced diversity and difficulty.

Abstract

Synthesizing high-quality instruction data from unsupervised text is a promising paradigm for training large language models (LLMs), yet automated methods for this task still exhibit significant limitations in the diversity and difficulty of synthesized instructions. To address these challenges, we propose Self-Foveate, an LLM-driven method for instruction synthesis. Inspired by hierarchical human visual perception, Self-Foveate introduces a "Micro-Scatter-Macro" multi-level foveation methodology that guides the extraction of textual information at three complementary granularities, from fine-grained details through cross-region connections to holistic patterns, thereby enhancing both the diversity and difficulty of synthesized instructions. Furthermore, a re-synthesis module is incorporated to improve the fidelity of instructions to source text and their overall quality. Comprehensive experiments across multiple unsupervised corpora and diverse model architectures demonstrate that Self-Foveate consistently outperforms existing methods. We publicly release our code at https://github.com/Mubuky/Self-Foveate.

Multi-Level Foveation Methodology

Inspired by hierarchical human visual perception, Self-Foveate introduces a "Micro-Scatter-Macro" methodology that extracts textual information at three complementary granularities:

Micro Level (Word): Fine-grained entity/attribute extraction focusing on individual words for detailed features.
Scatter Level (Multi-keyword): Cross-entity relationship grouping that combines 1-3 keywords into diverse feature groups.
Macro Level (Sentence): Rhetorical/figurative device extraction capturing complete sentences as contextual features.

Framework Overview

Compared to baseline methods like Self-QA that employ single-step generation producing simple and monotonous instruction candidates, Self-Foveate leverages multi-level foveation to:

Extract diverse details: Multi-level foveation enables the LLM to extract details (highlighted in distinct colors) of the text.
Synthesize with diversity: Different synthesis paradigms generate instructions with enhanced diversity and difficulty.
Ensure quality through re-synthesis: A re-synthesis module improves the fidelity of instructions to source text.

Experimental Results

Comprehensive experiments demonstrate that Self-Foveate consistently outperforms existing methods across multiple unsupervised corpora and diverse model architectures.

Accuracy trend across different settings

Recall trend across different settings

Downstream Task Performance

Recall (Rec.) and LLM Accuracy (Acc.) on downstream tasks: Self-Foveate vs. baselines.

Settings	GPT-4o mini						DeepSeek-V3
	SQuAD		HotpotQA		FilmWiki		SQuAD		HotpotQA		FilmWiki
	Rec.	Acc.	Rec.	Acc.	Rec.	Acc.	Rec.	Acc.	Rec.	Acc.	Rec.	Acc.
Llama-3.1-8B
None*	0.309	0.202	0.244	0.160	0.212	0.082	0.309	0.202	0.244	0.160	0.212	0.082
Self-QA	0.367	0.384	0.372	0.358	0.328	0.201	0.389	0.412	0.399	0.378	0.370	0.239
Wiki2023	0.327	0.361	0.338	0.322	0.333	0.235	0.342	0.370	0.340	0.328	0.349	0.244
Bonito*	0.386	0.405	0.360	0.372	0.219	0.153	0.386	0.405	0.360	0.372	0.219	0.153
Self-Foveate	0.484	0.490	0.507	0.486	0.512	0.367	0.481	0.491	0.525	0.501	0.548	0.397
Qwen2.5-7B
None*	0.251	0.300	0.266	0.234	0.139	0.032	0.251	0.300	0.266	0.234	0.139	0.032
Self-QA	0.249	0.232	0.276	0.246	0.206	0.082	0.119	0.125	0.102	0.106	0.111	0.056
Wiki2023	0.215	0.221	0.135	0.112	0.192	0.093	0.170	0.083	0.197	0.203	0.202	0.136
Bonito*	0.143	0.109	0.212	0.199	0.168	0.098	0.143	0.109	0.212	0.199	0.168	0.098
Self-Foveate	0.408	0.414	0.372	0.329	0.283	0.140	0.388	0.389	0.342	0.331	0.261	0.140
Gemma-2-9B
None*	0.224	0.121	0.175	0.078	0.211	0.099	0.224	0.121	0.175	0.078	0.221	0.099
Self-QA	0.383	0.409	0.408	0.389	0.429	0.315	0.402	0.435	0.424	0.408	0.509	0.386
Wiki2023	0.336	0.378	0.361	0.352	0.478	0.384	0.364	0.399	0.373	0.365	0.494	0.401
Bonito*	0.411	0.457	0.366	0.373	0.255	0.196	0.411	0.457	0.366	0.373	0.255	0.196
Self-Foveate	0.507	0.525	0.537	0.520	0.672	0.528	0.499	0.514	0.552	0.525	0.697	0.581

* Indicates that the base model was not fine-tuned using instructions synthesized by GPT-4o mini or DeepSeek-V3.

Citation

@inproceedings{li2025self,
  title={Self-Foveate: Enhancing Diversity and Difficulty of Synthesized Instructions from Unsupervised Text via Multi-Level Foveation},
  author={Li, Mingzhe and Lu, Xin and Zhao, Yanyan},
  booktitle={Findings of the Association for Computational Linguistics: ACL 2025},
  pages={7274--7289},
  year={2025}
}