Page MenuHomePhabricator
Files F65700735

Test tone-check retraining code locally
No One
Actions

Authored By
gkyziridis
Aug 1 2025, 3:39 PM
Size
10 KB
Referenced Files
None
Subscribers
None

Test tone-check retraining code locally
View Options

$ docker build --target production -f .pipeline/training/tone_check/retrain/blubber.yaml -t retrain:slim .
$ pwd
# ml-pipelines/training/tone_check
$ docker run --rm \
-v $(pwd)/data:/srv/edit_check/training/tone_check/data \
-v $(pwd)/base_model:/srv/edit_check/training/tone_check/base_model \
-v $(pwd)/output:/srv/edit_check/training/tone_check/output \
retrain:slim
# Output
INFO:root: -- Welcome to Tonecheck Retraining Job --
INFO:root:DEVICE: cpu
>>>> STARTED!!!
Generating train split: 3000 examples [00:00, 78989.27 examples/s]
INFO:root:train_dataset['train'][10]:
{'input': 'en[SEP]Peacock_Detection[SEP]they operate in several international markets . the company was founded in 1980 and has grown steadily since . this organization focuses on delivering quality services . he worked in finance and operations for over a decade . they operate in several international markets . this organization focuses on delivering quality services . the company was founded in 1980 and has grown steadily since . employees undergo regular training programs . he worked in finance and operations for over a decade . employees undergo regular training programs . the report was submitted for review . he worked in finance and operations for over a decade . the company was founded in 1980 and has grown steadily since . they operate in several international markets . employees undergo regular training programs . they operate in several international markets . employees undergo regular training programs . the company was founded in 1980 and has grown steadily since . he worked in finance and operations for over a decade . the system was updated to meet new regulatory requirements . the report was submitted for review . this organization focuses on delivering quality services . the company was founded in 1980 and has grown steadily since . this organization focuses on delivering quality services . they operate in several international markets . the report was submitted for review . the system was updated to meet new regulatory requirements . they operate in several international markets . they operate in several international markets . they operate in several international markets .', 'label': 0}
INFO:root:tokenizer loaded
DatasetDict({
train: Dataset({
features: ['input', 'label'],
num_rows: 2700
})
test: Dataset({
features: ['input', 'label'],
num_rows: 300
})
})
Map: 100%|██████████| 2700/2700 [00:00<00:00, 7525.25 examples/s]
Map: 100%|██████████| 300/300 [00:00<00:00, 7653.61 examples/s]
Filter: 100%|██████████| 2700/2700 [00:00<00:00, 6217.11 examples/s]
Filter: 100%|██████████| 300/300 [00:00<00:00, 5845.64 examples/s]
INFO:root:model is loaded: BertForSequenceClassification(
(bert): BertModel(
(embeddings): BertEmbeddings(
(word_embeddings): Embedding(119547, 768, padding_idx=0)
(position_embeddings): Embedding(512, 768)
(token_type_embeddings): Embedding(2, 768)
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
(encoder): BertEncoder(
(layer): ModuleList(
(0-11): 12 x BertLayer(
(attention): BertAttention(
(self): BertSdpaSelfAttention(
(query): Linear(in_features=768, out_features=768, bias=True)
(key): Linear(in_features=768, out_features=768, bias=True)
(value): Linear(in_features=768, out_features=768, bias=True)
(dropout): Dropout(p=0.1, inplace=False)
)
(output): BertSelfOutput(
(dense): Linear(in_features=768, out_features=768, bias=True)
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
)
(intermediate): BertIntermediate(
(dense): Linear(in_features=768, out_features=3072, bias=True)
(intermediate_act_fn): GELUActivation()
)
(output): BertOutput(
(dense): Linear(in_features=3072, out_features=768, bias=True)
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
)
)
)
)
(pooler): BertPooler(
(dense): Linear(in_features=768, out_features=768, bias=True)
(activation): Tanh()
)
)
(dropout): Dropout(p=0.1, inplace=False)
(classifier): Linear(in_features=768, out_features=2, bias=True)
)
INFO:root:Start training
0%| | 0/20 [00:00<?, ?it/s]/opt/lib/venv/lib/python3.11/site-packages/torch/utils/data/dataloader.py:665: UserWarning: 'pin_memory' argument is set as true but no accelerator is found, then device pinned memory won't be used.
warnings.warn(warn_msg)
20%|██ | 4/20 [00:28<01:37, 6.12s/it/opt/lib/venv/lib/python3.11/site-packages/transformers/configuration_utils.py:393: UserWarning: Some non-default generation parameters are set in the model config. These should go into either a) `model.generation_config` (as opposed to `model.config`); OR b) a GenerationConfig file (https://huggingface.co/docs/transformers/generation_strategies#save-a-custom-decoding-strategy-with-your-model).This warning will become an exception in the future.
Non-default generation parameters: {'max_length': 512}
warnings.warn(
/opt/lib/venv/lib/python3.11/site-packages/torch/utils/data/dataloader.py:665: UserWarning: 'pin_memory' argument is set as true but no accelerator is found, then device pinned memory won't be used.
warnings.warn(warn_msg)
40%|████ | 8/20 [00:54<01:10, 5.86s/it]{'eval_loss': 0.38032618165016174, 'eval_acc': 1.0, 'eval_roc_auc': 1.0, 'eval_threshold': 0.8366155624389648, 'eval_pr_auc': 1.0, 'eval_recall': 1.0, 'eval_precision': 1.0, 'eval_f1': 1.0, 'eval_tn': 2, 'eval_fp': 0, 'eval_fn': 0, 'eval_tp': 1, 'eval_runtime': 0.6659, 'eval_samples_per_second': 4.505, 'eval_steps_per_second': 1.502, 'epoch': 1.0}
40%|████ | 8/20 [00:54<01:10, 5.86s/it]/opt/lib/venv/lib/python3.11/site-packages/transformers/configuration_utils.py:393: UserWarning: Some non-default generation parameters are set in the model config. These should go into either a) `model.generation_config` (as opposed to `model.config`); OR b) a GenerationConfig file (https://huggingface.co/docs/transformers/generation_strategies#save-a-custom-decoding-strategy-with-your-model).This warning will become an exception in the future.
Non-default generation parameters: {'max_length': 512}
warnings.warn(
/opt/lib/venv/lib/python3.11/site-packages/torch/utils/data/dataloader.py:665: UserWarning: 'pin_memory' argument is set as true but no accelerator is found, then device pinned memory won't be used.
warnings.warn(warn_msg)
60%|██████ | 12/20 [01:20<00:47, {'eval_loss': 0.29208609461784363, 'eval_acc': 1.0, 'eval_roc_auc': 1.0, 'eval_threshold': 0.8407101035118103, 'eval_pr_auc': 1.0, 'eval_recall': 1.0, 'eval_precision': 1.0, 'eval_f1': 1.0, 'eval_tn': 2, 'eval_fp': 0, 'eval_fn': 0, 'eval_tp': 1, 'eval_runtime': 0.5785, 'eval_samples_per_second': 5.186, 'eval_steps_per_second': 1.729, 'epoch': 2.0}
60%|██████ | 12/20 [01:21<00:47, 5.91s/it/opt/lib/venv/lib/python3.11/site-packages/transformers/configuration_utils.py:393: UserWarning: Some non-default generation parameters are set in the model config. These should go into either a) `model.generation_config` (as opposed to `model.config`); OR b) a GenerationConfig file (https://huggingface.co/docs/transformers/generation_strategies#save-a-custom-decoding-strategy-with-your-model).This warning will become an exception in the future.
Non-default generation parameters: {'max_length': 512}
warnings.warn(
/opt/lib/venv/lib/python3.11/site-packages/torch/utils/data/dataloader.py:665: UserWarning: 'pin_memory' argument is set as true but no accelerator is found, then device pinned memory won't be used.
warnings.warn(warn_msg)
80%|████████ | 16/20 [01:47<00:23, 5.87s/it]{'eval_loss': 0.23636728525161743, 'eval_acc': 1.0, 'eval_roc_auc': 1.0, 'eval_threshold': 0.8521502614021301, 'eval_pr_auc': 1.0, 'eval_recall': 1.0, 'eval_precision': 1.0, 'eval_f1': 1.0, 'eval_tn': 2, 'eval_fp': 0, 'eval_fn': 0, 'eval_tp': 1, 'eval_runtime': 0.5762, 'eval_samples_per_second': 5.207, 'eval_steps_per_second': 1.736, 'epoch': 3.0}
80%|████████ | 16/20 [01:47<00:23, 5.87s/it/opt/lib/venv/lib/python3.11/site-packages/transformers/configuration_utils.py:393: UserWarning: Some non-default generation parameters are set in the model config. These should go into either a) `model.generation_config` (as opposed to `model.config`); OR b) a GenerationConfig file (https://huggingface.co/docs/transformers/generation_strategies#save-a-custom-decoding-strategy-with-your-model).This warning will become an exception in the future.
Non-default generation parameters: {'max_length': 512}
warnings.warn(
/opt/lib/venv/lib/python3.11/site-packages/torch/utils/data/dataloader.py:665: UserWarning: 'pin_memory' argument is set as true but no accelerator is found, then device pinned memory won't be used.
warnings.warn(warn_msg)
100%|██████████| 20/20 [02:13<00:00, 5.84s/it]{'eval_loss': 0.20355510711669922, 'eval_acc': 1.0, 'eval_roc_auc': 1.0, 'eval_threshold': 0.8665211796760559, 'eval_pr_auc': 1.0, 'eval_recall': 1.0, 'eval_precision': 1.0, 'eval_f1': 1.0, 'eval_tn': 2, 'eval_fp': 0, 'eval_fn': 0, 'eval_tp': 1, 'eval_runtime': 0.6483, 'eval_samples_per_second': 4.628, 'eval_steps_per_second': 1.543, 'epoch': 4.0}
100%|██████████| 20/20 [02:14<00:00, 5.84s/it/opt/lib/venv/lib/python3.11/site-packages/transformers/configuration_utils.py:393: UserWarning: Some non-default generation parameters are set in the model config. These should go into either a) `model.generation_config` (as opposed to `model.config`); OR b) a GenerationConfig file (https://huggingface.co/docs/transformers/generation_strategies#save-a-custom-decoding-strategy-with-your-model).This warning will become an exception in the future.
Non-default generation parameters: {'max_length': 512}
warnings.warn(
100%|██████████| 20/20 [02:16<00:00, 6.82s/it]
INFO:root:Trainning took: 136.44 secs
{'eval_loss': 0.19127391278743744, 'eval_acc': 1.0, 'eval_roc_auc': 1.0, 'eval_threshold': 0.8711633682250977, 'eval_pr_auc': 1.0, 'eval_recall': 1.0, 'eval_precision': 1.0, 'eval_f1': 1.0, 'eval_tn': 2, 'eval_fp': 0, 'eval_fn': 0, 'eval_tp': 1, 'eval_runtime': 0.5683, 'eval_samples_per_second': 5.279, 'eval_steps_per_second': 1.76, 'epoch': 5.0}
{'train_runtime': 136.3646, 'train_samples_per_second': 0.99, 'train_steps_per_second': 0.147, 'train_loss': 0.34897818565368655, 'epoch': 5.0}

File Metadata

Mime Type
text/plain; charset=utf-8
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
21624555
Default Alt Text
Test tone-check retraining code locally (10 KB)

Event Timeline

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits