AutoCompile
A compiler research system that trains language models to predict LLVM optimization sequences in one pass, without compiler feedback during inference.
Overview
Research / Compiler research
AutoCompile is a collaborative graduation research project that our team jointly designed, implemented, evaluated, and documented. It investigates whether fine-tuned language models can perform one-shot, tool-free LLVM pass-sequence prediction. Given LLVM IR and structural program features, the model predicts an optimization sequence in a single forward pass without compiler feedback. The system fine-tunes multiple LLMs using LoRA on offline bounded-search labels generated from a size-biased, exact-IR-deduplicated subset of 3,000 programs from IR-OptSet. Greedy N=1 inference achieves +1.15% guarded OverOz and exhibits template collapse, while best-of-40 sampling reaches +4.08%, exceeding the +3.92% bounded-search teacher ceiling — establishing that effective strategies exist within the model's distribution but are not reliably extracted under deterministic decoding.
AutoCompile was developed collaboratively, with responsibilities shared across the research pipeline rather than assigned as isolated individual components.
Technical stack
4 stack groups
ML & Fine-Tuning
7 instruments in this layerKey features
6 capability groups
Five-Stage SFT Pipeline
5 implementation notes- 01
Size-biased collection filters IR-OptSet to 3,000 programs with post-Oz instruction counts in [300, 10,000], targeting the large-function tail where optimization headroom is measurable (66% winner fraction).
- 02
Bounded autotuner generates teacher labels with budget B=100 opt evaluations per program, using a three-phase search: seed evaluation, random exploration, and hill climbing with mutation operators.
- 03
IR-fingerprint deduplication collapses exact-IR duplicates via SHA-1 hashing (after stripping comments, blanks, and whitespace normalisation) before splitting, with zero cross-split overlaps verified.
- 04
Stratified train/val/test split (80/10/10) allocates per OverOz bucket (baseline, small_win, medium_win, large_win) using the largest-remainder method to preserve headroom distribution.
- 05
LoRA fine-tuning (rank 16, α=16) on all seven projection matrices with response-only loss masking, early stopping with patience 3, and hard-fail guards on chat-template delimiters.
Architecture notes
6 recorded decisions
- 01
A five-stage offline SFT pipeline amortises a bounded per-program autotuner search (B=100) into a single forward-pass language model, eliminating inference-time compiler feedback.
- 02
IR-fingerprint deduplication using SHA-1 after whitespace normalisation, combined with stratified OverOz-bucket splitting, achieved zero cross-split leakage across 3,000 programs.
- 03
A feature-discriminability probe diagnosed greedy template collapse as a routing failure, with a random-forest classifier on 12 structural features reaching only 0.30 balanced accuracy for template prediction.
- 04
Controlled experiments showed that the N=1 to best-of-N gap persists across backbone scale, pretraining domain, GRPO reinforcement learning, and conditioning SFT.
- 05
A five-category reporting framework separates raw performance, guarded score, teacher reference, sampled capability, and correctness-verified results.
- 06
Response-only loss masking with hard-fail template guards concentrates training signal on the pass list rather than the much larger LLVM IR prefix.
Project status
2 status records
Completed Research
5 recorded items- 01
Full five-stage pipeline: size-biased collection, bounded autotuner, IR-fingerprint deduplication, stratified splitting, and LoRA fine-tuning.
- 02
Five controlled experiment variants: model-size ablation (3B/7B/14B), pretraining-prior ablation (Meta LLM Compiler), GRPO RL, and win-focused conditioning.
- 03
Best-of-40 inference exceeds the +3.92% teacher ceiling at +4.08% on the held-out test set (n=303), with 47 programs beating their individual teacher label.
- 04
Mechanistic explanation of greedy template collapse via Phase-0 feature-discriminability probe and input-information ceiling analysis.
- 05
Research paper with fourfold contribution: pipeline, gap localisation, collapse diagnosis, and five-category reporting framework.