Education and specialised fields
Language technology professional
The problem
A tokeniser that seems to work on formal sentences splits Albanian forms wrongly: contractions with an apostrophe, abbreviations with a full stop, numbers with thousands separators, or ë stored as two code points. The error flows unseen into lemmas and grammatical tags, where an analysis tool trained mostly on literary text also guesses at legal words it has never seen. Then the annotated corpus reports agreement counted token by token: the tokens outside any entity, which the two annotators nearly always agree on, inflate the score and hide the disagreement on the entities themselves.
26 courses · 1 specific to this profession
A process from your work
How long does it take you to label a document for training data?
- Step 1The tool places the first labels in the text.
- Step 2The expert reviews every label, adds the ones that are missing and deletes the wrong ones.
- Step 3The final label is the expert's, and the record shows it started from the tool.
- Where the tool goes wrongThe tool leaves some labels out and places others wrongly.
- The verification stepLabel a sample from scratch and compare it with the reviewed version.
More: what was measured and the full check
What changed, as measured
Starting from the model's pre-labels cut the time to produce expert labels by 58%, from 26.67 to 11.32 minutes per document.
Who was measured: Raters with medical backgrounds, labelling medication information in clinical notes, starting either from scratch or from the model's pre-labels.
The caveat: The authors work for the company that built the model; a single information extraction task on a single dataset; only human working time was counted.
The check, step by step
- Label a sample of documents from scratch, without the tool's labels, and compare it with the reviewed version.
- Count the labels added and the labels deleted separately: they show where the tool goes wrong.
- Never use the tool's labels as ground truth for evaluation; the test set is reviewed in full.
- Record in the data which labels started from the tool and who reviewed them.
The course covers processing Albanian text with Python: corpora, models and evaluation.
We can help you with this process: “Python and Albanian language processing”, 20 hours.
A documented failure pattern
94% accuracy, on letters seen in training
- What the tool doesA language technology specialist evaluates, for a ministry, a model that sorts incoming letters in Albanian by the directorate responsible. From the evaluation log, an AI tool drafts the summary of the report that will go to the ministry.
- Where the tool goes wrongThe summary reports accuracy but omits that some test letters were also in training.
- The verification stepRemove the test letters that were also in training and compute accuracy on the clean ones.
More: why it is easy to miss and the full check
Where the tool goes wrong, and why it is easy to miss
Three parts of the summary mislead the reader: "94%", "letters in Albanian" and "ready for use". The log shows that 640 of the 2,000 test letters were also in training, and that all of them come from the same archive for the same year; for those letters the model may remember the answer rather than work it out. The overlap line sits in the data description, not in the results, and the tool leaves it out. "Letters in Albanian" widens a test on one archive to all letters, and "ready for use" adds a conclusion the test did not measure. The figure itself is copied correctly, so a check against the results does not catch it. Published studies show that a model contaminated with test data looks better than it is, and that accuracy drops on new data.
The check that catches it
- Open the evaluation log and note, for the test set and for the training data, which source, which period and which directorate the letters come from.
- Search for matches between test and training, including near identical ones once dates, protocol numbers and names are removed, and take them out of the test set.
- Compute accuracy on the clean letters only and, where you can, on a new set from another year or directorate. Report both figures, not only the better one.
- In the summary, state the accuracy together with the set it was measured on, and delete any claim such as "ready for use" that the test does not support.
Where the failure pattern is documented: Academic studies: Findings of EMNLP 2023 (Association for Computational Linguistics, December 2023), the GSM1k study on arXiv (May 2024), the study on duplicate training data accepted to ACL 2022 (arXiv, July 2021).Sources: Findings of the Association for Computational Linguistics: EMNLP 2023, 2023; arXiv, 2024; arXiv, 2021.The case above is ours, written for work in Albania; the laws it names were checked against the official text.
What it looks like
The evaluation log"Test set: 2,000 letters from the same 2023 archive as the training data. Test letters also found in training, identical or near identical: 640. Accuracy on the test set: 94%."
The tool's summary"The model classifies letters in Albanian with 94% accuracy and is ready for use in all of the ministry's directorates."
Translated from the Albanian original.
Specialist courses for this profession
Python and Albanian language processing: working with texts, corpora and model evaluation.
What you leave with
The test set holds a case for every hard Albanian form (the apostrophe, abbreviations with a full stop, thousands separators, ë stored as two code points) with the expected split written by hand, and each fix in the code sits next to the case it protects; anyone who runs it gets the same result, pass or fail, for every case. With it go the error table, where each analysis error carries its type and root cause, and the annotation record, with agreement measured on spans and every adjudicated disagreement written down.
The workflow
You normalise the text, write the expected split by hand for each case, run the tokeniser and log every mismatch before changing the code; a fix is accepted only when the whole test set passes. Each analysis error is classified under the course's E1 to E5 taxonomy, with its root cause. Two annotators label the same sentences separately, and their agreement is measured on whole entity spans (start, end, label); every disagreement is adjudicated and written down, and the annotation guidelines are updated.
Language technology professional
The catalogue lists all 26 together, with filters for format and time.