| optimizer (AdamW) |
Adam with decoupled weight decay, standard for Transformer fine-tuning |
AdamW standard |
Use AdamW unless constrained; stable and well-supported |
| learning_rate |
Step size for weight updates |
1e-5 to 5e-5 (typical), occasionally 7e-5 with large batches |
Start at 2e-5 or 3e-5. Sweep {1e-5, 2e-5, 3e-5, 5e-5}. Scale up slightly with larger effective batch size |
| betas (β1, β2) |
Momentum terms for Adam |
(0.9, 0.999) |
Keep default; only adjust β2 to 0.98 if you observe very slow adaptation on small data |
| adam_epsilon |
Numerical stability in Adam |
1e-8 to 1e-6 |
For fp16/bf16, 1e-8 or 1e-6 both work; try 1e-8 first |
| weight_decay |
L2-like regularization (decoupled) |
0.0 to 0.1 (typical 0.01) |
0.01 is a strong default. Consider 0.02–0.05 if overfitting on very small datasets |
| lr_scheduler_type |
Shapes LR over steps |
linear, cosine, cosine_with_restarts, constant |
linear is a strong default; cosine may help stability for longer runs |
| warmup_ratio / warmup_steps |
Gradual LR ramp-up to stabilize early updates |
0.06–0.1 of total steps, or 100–2000 steps depending on run length |
Use warmup_ratio 0.06–0.1 for short fine-tunes; tune ±2% around a chosen ratio |
| num_train_epochs |
Total passes over training data |
2–5 typical; up to 10 on tiny datasets with early stopping |
Start at 3–4; add early stopping to avoid overfitting |
| per_device_train_batch_size |
Number of samples per device per step |
8–64 (memory-dependent) |
Choose the largest stable batch. If small batches, use gradient accumulation |
| gradient_accumulation_steps |
Accumulate grads to simulate larger batch |
1–8 |
Adjust to reach effective batch size 32–256 without OOM |
| effective_batch_size |
per_device_batch × accumulation × num_devices |
32–256 effective |
For LR scaling, roughly linearly scale LR with effective batch size within reasonable bounds |
| max_grad_norm |
Gradient clipping to prevent exploding grads |
0.5–1.0 (common 1.0) |
Keep at 1.0; lower to 0.5 if you see instability spikes |
| eval_steps / evaluation_strategy |
Validation frequency for early stopping |
every 100–1000 steps or per epoch |
Higher frequency for small datasets to react early to overfitting |
| seed |
Controls initialization and data shuffling |
integer |
Run 3–5 seeds to report mean and std; fix seed for final model |
| mixed_precision |
Faster training with reduced precision |
fp16 or bf16 (hardware dependent) |
Use bf16 if supported; otherwise fp16. Monitor loss scaling if using fp16 |
| gradient_checkpointing |
Trades compute for memory |
on/off |
Enable when memory-limited; expect slower wall-clock |
| max_seq_length |
Maximum tokenized length |
128–256 typical; 512 for long reviews |
Prefer 128 for short Chinese sentiment tasks to maximize throughput; profile 256 if texts are longer |
| padding/truncation strategy |
How sequences are padded/truncated |
dynamic padding; truncation=longest_first |
Use dynamic padding for efficiency; use longer_max_length only if accuracy requires it |
| attention_dropout (encoder) |
Dropout on attention probabilities |
0.1 (BERT default) |
Usually keep default; can raise to 0.15 on very small data if reloading with config override |
| hidden_dropout_prob (encoder) |
Dropout on hidden states |
0.1 (BERT default) |
Keep at 0.1; 0.1–0.2 if small data and you reload with modified config |
| classifier_dropout |
Dropout before classification head |
0.1–0.3 |
0.1 default; try 0.2–0.3 on small/noisy datasets |
| label_smoothing_factor |
Softens hard labels to reduce overconfidence |
0.0–0.1 |
0.05 is a good starting value for small or noisy data |
| class_weights |
Reweights classes for imbalance |
proportional to 1/freq |
Use when class imbalance harms macro-F1. Validate that loss decreases stably |
| freeze_encoder_layers |
Freeze early Transformer layers |
0–10 layers |
Consider freezing 6–10 layers on very small data to reduce overfitting and speed training |
| layerwise_lr_decay (LLRD) |
Decrease LR for lower layers |
0.8–0.95 decay factor |
Start at 0.95; combine with slightly higher top LR (e.g., head ×2 LR) |
| reinit_top_layers |
Re-initialize top k Transformer layers |
0–2 |
Try 1–2 on small data to reduce catastrophic forgetting; stabilize with warmup |
| pooling_strategy |
CLS vs mean pooling |
CLS (default), mean |
CLS is standard; mean pooling can help on noisy short texts—benchmark both |
| num_labels |
Number of sentiment classes |
2 or 3 typically |
Ensure consistent loss/metrics; reinitialize classification head when changed |
| early_stopping_patience |
Stop if no val improvement |
2–3 eval checks |
Use with moving average metrics; monitor macro-F1 if classes are imbalanced |
| dataloader_num_workers |
Parallelism for data loading |
2–8 (system-dependent) |
Tune for throughput; avoid CPU bottlenecks |
| fp16/bf16 loss scaling |
Stabilizes mixed-precision training |
dynamic/static |
Use dynamic scaling (framework default) to avoid underflow |
