Zone detection model for (17th–18th c.) manuscripts, trained on Leibniz's manuscripts using SegmOnto ontology - V2.
This model was developed by Denisa-Florina BUMBA as part of the ERC Philiumm project (Grant ADG No. 101020985), at the Laboratoire SPHERE, Université Paris Cité – CNRS, France.
Fine-tuned RF-DETR-Seg (Nano) for instance segmentation and zone detection on scanned pages of Leibniz's manuscripts. The model detects zones such as main text, margin text, numbering, diagrams, digitization artifacts, and several classes of mathematical formulae using the SegmOnto controlled vocabulary, in order to support automatic transcription in the digital humanities. It is trained specifically on Leibniz's manuscripts, but may be applied to other historical manuscripts with similar layout.
Previous version: v1
What's new in v2 :
- corrected annotations, particularly for mathematical formulae zones.
- increased the training resolution from 720 px to 1272 px.
Classes
| Class | Description |
|---|---|
MainZone |
Main body text region |
MarginTextZone |
Marginal additions |
NumberingZone |
Page numbering |
DigitizationArtefactZone |
Digitization artefacts |
GraphicZone-figure |
Diagrams, other graphic zones |
MathsRegion-complex |
Complex/multi-line formulae |
MathsRegion-inline |
Inline formulae within text |
MathsRegion-strikethrough |
Struck-through/deleted formulae |
Training Data
| Split | Images | Percentage |
|---|---|---|
| Train | 1,002 | 94% |
| Validation | 40 | 4% |
| Test | 18 | 2% |
| Total | 1,060 | 100% |
Performance
Results on the validation set.
| Checkpoint | Epoch | segm mAP@50:95 |
|---|---|---|
checkpoint_best_regular_v2.pth |
40/60 | 0.4670 |
checkpoint_best_ema_v2.pth |
40/60 | 0.4700 |
Per-class validation results for EMA checkpoint v2.
| Class | AP |
|---|---|
| DigitizationArtefactZone | 0.9380 |
| MainZone | 0.9117 |
| MarginTextZone | 0.6194 |
| NumberingZone | 0.4945 |
| GraphicZone-figure | 0.5552 |
| MathsRegion-complex | 0.1913 |
| MathsRegion-inline | 0.1280 |
| MathsRegion-strikethrough | 0.0752 |
How to Get Started with the Model
from rfdetr import RFDETRSegNano
import cv2
from rfdetr.assets.coco_classes import COCO_CLASSES
model = RFDETRSegNano(pretrain_weights="checkpoint_best_regular_v2.pth")
image = cv2.imread("image.jpg")
detections = model.predict(image, threshold=0.4)
labels = [f"{COCO_CLASSES[class_id]}" for class_id in detections.class_id]
annotated_image = sv.MaskAnnotator().annotate(detections.metadata["source_image"], detections)
annotated_image = sv.LabelAnnotator().annotate(annotated_image, detections, labels)
cv2.imwrite("image_annotated.jpg", annotated_image)
Replace "image.jpg" with the path to your manuscript image.
The threshold can be changed, but we noticed that we have fair predictions with a value of 0,4. Under this value we start having overlapping regions and over this value we loose some correct predictions with low confidence score.
eScriptorium pipeline
A preprocessing pipeline for preparing model outputs for import into eScriptorium is available here:
RF-DETR-Seg to XML Page pipeline
Funding
This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme, Advanced Grant ADG No. 101020985.