Repository navigation
Sync: Develop to main - #51
Merged
Merged
Conversation
…er base_model_prefix LlamaForQuestionAnswering and Qwen2ForQuestionAnswering keep base_model_prefix "transformer" for backward compatibility, but their checkpoints store the backbone under "model.*". AutoModelForQuestionAnswering therefore reported the whole backbone as newly initialized and extractive QA fine-tuning of decoder models started from random weights (SQuAD F1 around 20 instead of 65+). Load through a helper that inspects loading_info, retries with base_model_prefix="model" when only that mismatch is present, and raises if any backbone tensor is still missing.
…eight loading Encoder QA heads such as ModernBertForQuestionAnswering add classifier/head parameters that are expected to be newly initialized. Classify missing keys by the model's base_model_prefix instead of everything outside qa_outputs.
Fix decoder backbone loading in extractive QA evaluation
Align the 3B English pre-training budget with the 1B design (stage1 + stage2 ~= 20x non-embedding parameters, same epoch for Llama and ModernBERT within a language): - stage1 (FineWeb-Edu, seq 1024): 0.45 -> 0.65 epoch Llama-EN-3B 16,270 steps / 68.2B tokens, ModernBERT-EN-3B 13,892 steps / 58.3B tokens - stage2 (wiki40b, seq 8192): 3 -> 1 epoch (same as 1B) ~2.5B tokens each Totals: Llama-EN-3B 70.7B (25x of 2.83B), ModernBERT-EN-3B 60.8B (19.9x of 3.05B). JP configs are unchanged (0.8 / 6 epochs).
…ge2-1epoch Raise EN 3B stage1 to 0.65 epoch and cut EN 3B stage2 to 1 epoch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WHY
WHAT