Page MenuHomePhabricator
Paste P80443

Test tone-check retraining code locally
ActivePublic
Actions

Authored by gkyziridis on Aug 1 2025, 3:39 PM.
Tags
None
Referenced Files
F65700735: Test tone-check retraining code locally
Aug 1 2025, 3:39 PM
Subscribers
None
$ docker build --target production -f .pipeline/training/tone_check/retrain/blubber.yaml -t retrain:slim .
$ pwd
# ml-pipelines/training/tone_check
$ docker run --rm \
-v $(pwd)/data:/srv/edit_check/training/tone_check/data \
-v $(pwd)/base_model:/srv/edit_check/training/tone_check/base_model \
-v $(pwd)/output:/srv/edit_check/training/tone_check/output \
retrain:slim
# Output
INFO:root: -- Welcome to Tonecheck Retraining Job --
INFO:root:DEVICE: cpu
>>>> STARTED!!!
Generating train split: 3000 examples [00:00, 78989.27 examples/s]
INFO:root:train_dataset['train'][10]:
{'input': 'en[SEP]Peacock_Detection[SEP]they operate in several international markets . the company was founded in 1980 and has grown steadily since . this organization focuses on delivering quality services . he worked in finance and operations for over a decade . they operate in several international markets . this organization focuses on delivering quality services . the company was founded in 1980 and has grown steadily since . employees undergo regular training programs . he worked in finance and operations for over a decade . employees undergo regular training programs . the report was submitted for review . he worked in finance and operations for over a decade . the company was founded in 1980 and has grown steadily since . they operate in several international markets . employees undergo regular training programs . they operate in several international markets . employees undergo regular training programs . the company was founded in 1980 and has grown steadily since . he worked in finance and operations for over a decade . the system was updated to meet new regulatory requirements . the report was submitted for review . this organization focuses on delivering quality services . the company was founded in 1980 and has grown steadily since . this organization focuses on delivering quality services . they operate in several international markets . the report was submitted for review . the system was updated to meet new regulatory requirements . they operate in several international markets . they operate in several international markets . they operate in several international markets .', 'label': 0}
INFO:root:tokenizer loaded
DatasetDict({
train: Dataset({
features: ['input', 'label'],
num_rows: 2700
})
test: Dataset({
features: ['input', 'label'],
num_rows: 300
})
})
Map: 100%|██████████| 2700/2700 [00:00<00:00, 7525.25 examples/s]
Map: 100%|██████████| 300/300 [00:00<00:00, 7653.61 examples/s]
Filter: 100%|██████████| 2700/2700 [00:00<00:00, 6217.11 examples/s]
Filter: 100%|██████████| 300/300 [00:00<00:00, 5845.64 examples/s]
INFO:root:model is loaded: BertForSequenceClassification(
(bert): BertModel(
(embeddings): BertEmbeddings(
(word_embeddings): Embedding(119547, 768, padding_idx=0)
(position_embeddings): Embedding(512, 768)
(token_type_embeddings): Embedding(2, 768)
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
(encoder): BertEncoder(
(layer): ModuleList(
(0-11): 12 x BertLayer(
(attention): BertAttention(
(self): BertSdpaSelfAttention(
(query): Linear(in_features=768, out_features=768, bias=True)
(key): Linear(in_features=768, out_features=768, bias=True)
(value): Linear(in_features=768, out_features=768, bias=True)
(dropout): Dropout(p=0.1, inplace=False)
)
(output): BertSelfOutput(
(dense): Linear(in_features=768, out_features=768, bias=True)
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
)
(intermediate): BertIntermediate(
(dense): Linear(in_features=768, out_features=3072, bias=True)
(intermediate_act_fn): GELUActivation()
)
(output): BertOutput(
(dense): Linear(in_features=3072, out_features=768, bias=True)
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
)
)
)
(pooler): BertPooler(
(dense): Linear(in_features=768, out_features=768, bias=True)
(activation): Tanh()
)
)
(dropout): Dropout(p=0.1, inplace=False)
(classifier): Linear(in_features=768, out_features=2, bias=True)
)
INFO:root:Start training
0%| | 0/20 [00:00<?, ?it/s]/opt/lib/venv/lib/python3.11/site-packages/torch/utils/data/dataloader.py:665: UserWarning: 'pin_memory' argument is set as true but no accelerator is found, then device pinned memory won't be used.
warnings.warn(warn_msg)
20%|██ | 4/20 [00:28<01:37, 6.12s/it/opt/lib/venv/lib/python3.11/site-packages/transformers/configuration_utils.py:393: UserWarning: Some non-default generation parameters are set in the model config. These should go into either a) `model.generation_config` (as opposed to `model.config`); OR b) a GenerationConfig file (https://huggingface.co/docs/transformers/generation_strategies#save-a-custom-decoding-strategy-with-your-model).This warning will become an exception in the future.
Non-default generation parameters: {'max_length': 512}
warnings.warn(
/opt/lib/venv/lib/python3.11/site-packages/torch/utils/data/dataloader.py:665: UserWarning: 'pin_memory' argument is set as true but no accelerator is found, then device pinned memory won't be used.
warnings.warn(warn_msg)
40%|████ | 8/20 [00:54<01:10, 5.86s/it]{'eval_loss': 0.38032618165016174, 'eval_acc': 1.0, 'eval_roc_auc': 1.0, 'eval_threshold': 0.8366155624389648, 'eval_pr_auc': 1.0, 'eval_recall': 1.0, 'eval_precision': 1.0, 'eval_f1': 1.0, 'eval_tn': 2, 'eval_fp': 0, 'eval_fn': 0, 'eval_tp': 1, 'eval_runtime': 0.6659, 'eval_samples_per_second': 4.505, 'eval_steps_per_second': 1.502, 'epoch': 1.0}
40%|████ | 8/20 [00:54<01:10, 5.86s/it]/opt/lib/venv/lib/python3.11/site-packages/transformers/configuration_utils.py:393: UserWarning: Some non-default generation parameters are set in the model config. These should go into either a) `model.generation_config` (as opposed to `model.config`); OR b) a GenerationConfig file (https://huggingface.co/docs/transformers/generation_strategies#save-a-custom-decoding-strategy-with-your-model).This warning will become an exception in the future.
Non-default generation parameters: {'max_length': 512}
warnings.warn(
/opt/lib/venv/lib/python3.11/site-packages/torch/utils/data/dataloader.py:665: UserWarning: 'pin_memory' argument is set as true but no accelerator is found, then device pinned memory won't be used.
warnings.warn(warn_msg)
60%|██████ | 12/20 [01:20<00:47, {'eval_loss': 0.29208609461784363, 'eval_acc': 1.0, 'eval_roc_auc': 1.0, 'eval_threshold': 0.8407101035118103, 'eval_pr_auc': 1.0, 'eval_recall': 1.0, 'eval_precision': 1.0, 'eval_f1': 1.0, 'eval_tn': 2, 'eval_fp': 0, 'eval_fn': 0, 'eval_tp': 1, 'eval_runtime': 0.5785, 'eval_samples_per_second': 5.186, 'eval_steps_per_second': 1.729, 'epoch': 2.0}
60%|██████ | 12/20 [01:21<00:47, 5.91s/it/opt/lib/venv/lib/python3.11/site-packages/transformers/configuration_utils.py:393: UserWarning: Some non-default generation parameters are set in the model config. These should go into either a) `model.generation_config` (as opposed to `model.config`); OR b) a GenerationConfig file (https://huggingface.co/docs/transformers/generation_strategies#save-a-custom-decoding-strategy-with-your-model).This warning will become an exception in the future.
Non-default generation parameters: {'max_length': 512}
warnings.warn(
/opt/lib/venv/lib/python3.11/site-packages/torch/utils/data/dataloader.py:665: UserWarning: 'pin_memory' argument is set as true but no accelerator is found, then device pinned memory won't be used.
warnings.warn(warn_msg)
80%|████████ | 16/20 [01:47<00:23, 5.87s/it]{'eval_loss': 0.23636728525161743, 'eval_acc': 1.0, 'eval_roc_auc': 1.0, 'eval_threshold': 0.8521502614021301, 'eval_pr_auc': 1.0, 'eval_recall': 1.0, 'eval_precision': 1.0, 'eval_f1': 1.0, 'eval_tn': 2, 'eval_fp': 0, 'eval_fn': 0, 'eval_tp': 1, 'eval_runtime': 0.5762, 'eval_samples_per_second': 5.207, 'eval_steps_per_second': 1.736, 'epoch': 3.0}
80%|████████ | 16/20 [01:47<00:23, 5.87s/it/opt/lib/venv/lib/python3.11/site-packages/transformers/configuration_utils.py:393: UserWarning: Some non-default generation parameters are set in the model config. These should go into either a) `model.generation_config` (as opposed to `model.config`); OR b) a GenerationConfig file (https://huggingface.co/docs/transformers/generation_strategies#save-a-custom-decoding-strategy-with-your-model).This warning will become an exception in the future.
Non-default generation parameters: {'max_length': 512}
warnings.warn(
/opt/lib/venv/lib/python3.11/site-packages/torch/utils/data/dataloader.py:665: UserWarning: 'pin_memory' argument is set as true but no accelerator is found, then device pinned memory won't be used.
warnings.warn(warn_msg)
100%|██████████| 20/20 [02:13<00:00, 5.84s/it]{'eval_loss': 0.20355510711669922, 'eval_acc': 1.0, 'eval_roc_auc': 1.0, 'eval_threshold': 0.8665211796760559, 'eval_pr_auc': 1.0, 'eval_recall': 1.0, 'eval_precision': 1.0, 'eval_f1': 1.0, 'eval_tn': 2, 'eval_fp': 0, 'eval_fn': 0, 'eval_tp': 1, 'eval_runtime': 0.6483, 'eval_samples_per_second': 4.628, 'eval_steps_per_second': 1.543, 'epoch': 4.0}
100%|██████████| 20/20 [02:14<00:00, 5.84s/it/opt/lib/venv/lib/python3.11/site-packages/transformers/configuration_utils.py:393: UserWarning: Some non-default generation parameters are set in the model config. These should go into either a) `model.generation_config` (as opposed to `model.config`); OR b) a GenerationConfig file (https://huggingface.co/docs/transformers/generation_strategies#save-a-custom-decoding-strategy-with-your-model).This warning will become an exception in the future.
Non-default generation parameters: {'max_length': 512}
warnings.warn(
100%|██████████| 20/20 [02:16<00:00, 6.82s/it]
INFO:root:Trainning took: 136.44 secs
{'eval_loss': 0.19127391278743744, 'eval_acc': 1.0, 'eval_roc_auc': 1.0, 'eval_threshold': 0.8711633682250977, 'eval_pr_auc': 1.0, 'eval_recall': 1.0, 'eval_precision': 1.0, 'eval_f1': 1.0, 'eval_tn': 2, 'eval_fp': 0, 'eval_fn': 0, 'eval_tp': 1, 'eval_runtime': 0.5683, 'eval_samples_per_second': 5.279, 'eval_steps_per_second': 1.76, 'epoch': 5.0}
{'train_runtime': 136.3646, 'train_samples_per_second': 0.99, 'train_steps_per_second': 0.147, 'train_loss': 0.34897818565368655, 'epoch': 5.0}
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits