Data · dataset · 2024
POS-tagging model: Marmot
Listed in Researchdata.se
Model Marmot is a part-of-speech tagger that performs for Swedish somewhat worse than state-of-the-art neural models, but still yields very good results, works much faster and does not require a GPU.
Description
We provide two models for Marmot. marmot_eval is trained on SUC3 and Talbanken_SBX_dev, using Saldo as dictionary. The advantage of this model is that it can be evaluated, using Talbanken_SBX_test or SIC2.
The evaluation results are reported in the table below. Test set Exact match POS MSD Talbanken_SBX_test 0.973 0.982 0.988 SIC2 0.921 0.934 0.958 Read more about the evaluation here. marmot_full is trained on SUC3 + Talbanken_SBX_test + Talbanken_SBX_dev + SIC2 (with Saldo as dictionary). We cannot evaluate the performance of this model, but we expect it to perform better than marmot_eval, or at least not worse.
Read the rest (3 more)
Tagging and training Download Marmot and the necessary dependencies. Download SALDO (converted to the necessary format) here. Download our scripts from this repository.
The scripts use a tab-separated three-column format: token, POS (without MSD), MSD. Use conllu_to_tab.rb to convert CONLL(U) to the two-column format (install Ruby 1.9+ and run ruby conllu_to_tab 2 n, where n is the number of the column you want to use (if you are converting our CONLLU files, use 4). Run ruby convert_col2_to_marmot.rb to convert the resulting col2 file to Marmot's col3).
Tagging Use java -cp marmot.jar marmot.morph.cmd.Annotator --model-file model_name.marmot --test-file form-index=0,test_corpus.col1 --pred-file output_name.conll to tag a corpus using a pretrained model. The output corpus will be in a CONLL format with a somewhat unusual order of columns, use convert_marmot_to_conllu.rb to convert it to a usual CONLLU. Training your own models Run Marmot: java -Xmx5G -cp marmot.jar marmot.morph.cmd.Trainer -train-file form-index=0,tag-index=1,morph-index=2,corpus.col3 -tag-morph true -model-file model_name.marmot subtag-separator "." -type-dict saldo_marmot.txt,indexes=[2,3]
Links
Where it is published
- DOI doi.org/10.23695/aryw-nh78 ↗
DOI / persistent id · from researchdata se
Catalogue records · 1
- OAI-PMH record api.researchdata.se/oai-pmh?verb=GetRecord&metadataPrefix=oai_dc&identifier=oai%3A… ↗
metadata API · from researchdata se
Topics
- From keywords
- Computer Science & AI · Earth & Environmental Science · Humanities
- Inferred from text
- Tabular 65%
Provenance · 1 source records, 9 field assertions
| Source | Key | Last seen | Raw |
|---|---|---|---|
| Researchdata.se | oai:researchdata.se:doi-10-23695-aryw-nh78/0 | 10 d ago | JSON v1 |
| Field | Assertion | Extractor | Evidence |
|---|---|---|---|
| access_level | source · researchdata se | connector:researchdata_se@1.0.0 | |
| concepts[field].local:field:computer-science-ai | mapping · researchdata se | connector:researchdata_se@1.0.0 | |
| concepts[field].local:field:earth-environmental | mapping · researchdata se | connector:researchdata_se@1.0.0 | |
| concepts[field].local:field:humanities | mapping · researchdata se | connector:researchdata_se@1.0.0 | |
| concepts[modality].local:modality:tabular | enrichment · researchdata se | keyword-concept-rules@1.0.0 | title+description (65%) |
| description | source · researchdata se | connector:researchdata_se@1.0.0 | /metadata/dc/description |
| license | source · researchdata se | connector:researchdata_se@1.0.0 | /metadata/dc/rights |
| publication_date | source · researchdata se | connector:researchdata_se@1.0.0 | |
| title | source · researchdata se | connector:researchdata_se@1.0.0 | /metadata/dc/title |