Research Protocol
Bibliometric network analysis of Viking Age research (BibVik)
Note that the project has significantly evolved since drafting this protocol, and this document does not reflect our current scope of work. It still has value as a marker of our original intent.
1 Overview and Scope
This project seeks to identify trends in the published archaeological literature corresponding with shifting attitudes concerning the origins of the Viking Age, as initially surmised by Lund and Sindbæk (2022).
[Why is this important?]
More specifically, we ask:
- How do various perspectives on the origins of the Viking Age manifest in the publication record?
- Are these distinct perspectives rooted in personal or professional identities and/or relationships?
- Are these distinct perspectives informed by different kinds of sources to inform their work and/or back their claims?
1.1 Predictions and Expectations
Here we document some predicted outcomes, which may or may not come to fruition. However, it is good to express our assumptions and expectations at this early point, for purposes of reflexivity.
- We are going to find some sub-clusters within clusters, and perhaps also “bridging” clusters.
- Some sub-clusters will be tight-knit and closed, others will be broad and open.
- Some clusters will be focused on specific topics.
- There will be many links across institutions, they are not really that fixed.
- Some citation chains will emerge that reflect casual observations (just a few examples):
- Lots of cross-references within the Mälaren valley community.
- Tight-knit cluster among papers that are concerned with ritual.
- Major computational cluster at Aarhus.
- If we had a broader time horizon, we might be able to trace “genealogies of affiliation” by tracing how identities change based on the evolution of their collaborative networks.
- Clustering of methods by “academic upbringing”, which has a localized character.
- We may be able to find associations based on age, but mainly by doing a rough assessment of the n most and least central nodes, rather than keying age into the data.
2 Materials and Methods
This project is based on a mixed-methods (qualitative-quantitative) analysis. The general idea is to qualitatively annotate and describe a corpus of published papers, and then to quantitatively explore the relationships between papers and their properties through exploratory data analysis (following Tukey 1977) and network analysis methods. This follows a similar approach as Batist and Roe (2024).
This requires us to collect and enhance information about a corpus of relevant publications, the relationships among them, and the social contexts through which they were produced.
2.1 The corpus of works
This project is fundamentally rooted on analysis of published papers, reports and proceedings, which we consider representative of a broader discursive phenomenon.
The corpus of works is rooted in Lund and Sindbæk (2022), which is a review that… [summarize the paper in one or two sentences here]. See the supplement for information on alternative (and less suitable) methods for producing a corpus that we decided not to pursue.
Aside from its representative content, Lund and Sindbæk (2022) is a great seed paper due to the circumstances of its creation. It is authored by two eminent archaeologists of the Viking Age, who are intimately familiar with the community of which they wrote. The authors also acknowledge their adherence to the different fields they identify, which imbues the paper with a balanced perspective. Moreover, the paper, which was published in the Journal of Archaeological Science, underwent an unusually rigorous peer-review process through which seven anonymous reviewers recommended additional sources and commentary on their respective character.1 Overall, the seed paper is a summary of a discursive phenomenon led by two extremely experienced archaeologists who represent different yet converging perspectives, which has been supplemented and validated by a jury of seven anonymous peers.
The corpus will emerge from this seed paper through its bibliographic references, which will be identified using CrossRef or OpenCitations. Using a genealogical metaphor, the seed paper may be referred to as P (for parental generation), its references as F1 (first filial generation), their bibliographic references as F2 (second filial generation), and so on. These linkages will enable us to ascertain whether the chains of references that each paper relies on follow any identifiable patterns, amalgamate into any discrete clusters, or point toward any “common ancestors” from which new lineages emerged or diverged.
2.2 Annotating papers
This formal genealogical approach will be greatly enhanced through qualitative assessment of each paper. Ascertaining the qualities of each paper will enable us to understand their respective roles in the corpus rather than their mere presence.
A significant portion of work will therefore involve identifying, characterizing and annotating the series of papers that are referenced in the seed paper. This will entail extracting structured information regarding the papers’ authors and publication venues, and characterizing each paper by kind (i.e. reports on findings, reviews of prior work, opinion pieces, etc). We will then determine the character of each paper, as considered in Lund and Sindbæk (2022), and through an additional qualitative assessment of each paper following an a priori coding protocol (methodological details presented below).
It should be noted that we will only be able to annotate papers referenced in the seed paper. In other words, our qualitative evaluation is limited to a single generation. However, we may sample the F1 generation and conduct deeper qualitative analysis based on sampled subsets.
2.3 Relational and contextual ties
We will also include additional information to help contexualize each paper as a product of social and situational circumstances. This will involve retrieving additional metadata about the papers, including the journals or venues in which they were published, as well as information about their authors, including their affiliations, genders and areas of expertise.2 Much of this information may be retrieved from CrossRef.
This information will allow us to ascertain a few key insights regarding publishing and citation practices. For instance, we may observe that certain kinds of work tend to be published in edited volumes rather than as standalone papers, to have specific institutional origins, to have authors of a specific gender. Aside from documenting the nature of discrete clusters of works, these factors may contribute to greater consideration of citational politics.
Regarding gender, we may use the gender R package, created specifically for this purpose. The information assigned through both of these processes will have to be manually verified, but the community is small enough to make this feasible. We may consult with community interest groups (e.g. the SAA Queer Archaeology Interest group) to help support or provide critical feedback on “best practices” for identifying gender or other identity-based information.
It is important to be clear about our intent, since this influences (a) the ways we go about tryjng to achieve our goals, and (b) the reception of our approach in relation to contemporary discourse on gender diversity in research. Out goal is not to identify gender diversity or to assign gendered characteristics to certain clusters or subclusters, and these objectives would not be supported by our methods and supporting data. Instead, our intent is to identify citation biases and to critically assess the citational politics of research on the emergence of the Viking Age.
3 Analysis
A significant component of this work will involve network analysis to identify insular or bridging clusters of published works, the characteristic properties of these clusters, as well as common ancestors from which discrete clusters emerged or diverged.
One relatively simple form of analysis could involve validating the conclusions arrived at by Lund and Sindbæk (2022). Specifically, we may determine whether or not two distinct schools of thought actually exist, or whether there is more overlap or nuance apparent in the data. Information about authorship and publication venues may also accompany a more detailed breakdown of the character of each cluster.
Additionally, the directed nature of the relationships will enable us to ascertain pathways through which ideas flowed over time. Since each relationship will be tagged as a supporting, contrasting/disputing or mentioning citation, we may be able to determine whether certain papers are loci of agreement or conflict through the proportions atittudes with which papers cite them (and whether those attitudes correspond with certain clusters).
We will sure realize more analytical opportunities as the work progresses, and we will document these insights here as they arise.
4 Methodological notes and ideas
4.1 Qualitative methods
A research assistant will annotate each reference using a series of keywords deriving from a controlled vocabulary. The controlled vocabulary has three axes:
- Topics
- What is the paper about?
- i.e. diaspora, mobility, gender, ritual, etc
- Methods
- What methods were applied in the paper?
- i.e. spatial analysis, genetics, stable isotope analysis, finds analysis
- Kind of contribution
- i.e. field report, “standard” scientific paper, review of prior work, opinion, etc
- Medium
- i.e. journal article, book, chapter in an edited volume, etc
The research assistant will identify and record key terms from the controlled vocabulary for each paper, based on their reading of the abstract, introduction, conclusion or other summaritive section. Since this does not involve comprehensive or open coding, this can simply be entered into a customized spreadsheet, rather than using some specialized QDA software.
The research assistant will ideally be a more experienced student in the field of Viking Age archaeology, who may also benefit from this as a learning experience. Zack will schedule weekly check-ins with the research assistant and make himself available to address any methodological questions or concerns. Julie will then verify the annotations after the research assistant has completed the work.
4.2 Analyzing references
We may characterize the citations themselves to ascertain the nature of each reference. We may follow a relatively simple classification scheme, i.e. supporting, contrasting/disputing or mentioning, which can be determined using natural language processing (NLP) (specifically, the scite zotero plugin). Alternatively, we may qualitatively code each reference following a more specialized coding scheme, as per Huvila, Andersson, and Sköld (2022) (see additional notes on this approach below).
However, upon further reflection, this may not be feasible given the project’s timeframe, our material resources, and the reliability of the software solutions at hand. Qualitative coding can be very time-consuming, and we are only able to hire an assistant for four weeks of work to analyze over 500 references (in addition to other research tasks). Moreover, scite tends to “play it safe” by coding references as neutral mentions when human coders identify a more robust stance (see Bakker, Theis-Mahon, and Brown 2023). Additionally, scite’s corpus is limited to works put out by partnered publishers. There is an open source alternative to scite (see Jurgens (2017); https://jurgens.people.si.umich.edu/citation-function) but it requires more knowledge of NLP methods than we are able to implement.
These challenges aside, it may be reasonable to derive a sample based on network statistics, and then qualitatively annotate the references within the sample.
4.3 Statistical analysis
Keeping track of topics independently from citational links enables us to validate whether topics are actually clustered around certain individuals or papers, or whether there are diverse centres.
We’ll be able to do some non-network descriptive statistics:
- Distribution of topics by affiliations of authors.
- Distribution of topics by number of co-authors.
- Temporal distributions of genders of lead authors.
- Gender of lead author by size and genders of their co-authors.
4.4 AI tools and resources
We may apply LLM, NLP or topic-modelling methods to simplify, verify, or compare how it differs from human-led classification of research papers.
MaxQDA has some guidelines on how to use AI to enhance literature reviews. However it’s unclear whether any effort has been made to specialize the use of AI toward lit reviews specifically. I’m sure there are additional services that specialize in AI-assisted literature reviews too.
Topic modelling??
I wonder if we can relate this work to Alex Brandsen’s dissertation research (Brandsen and Lippok 2021; Brandsen 2022, 2023).
4.5 Alternative means of assembling the corpus
There are a few alternative approaches we may take to collect different kinds of datasets.
One approach is to conduct systematic keyword searches to return a subset of the archaeological literature for deeper evaluation, using a tool like Publish or Perish or the various R packages designed to access scholarly reference databases (e.g. rcrossref, wosr, openalexR, citecorp). This affords a relatively systematic approach to literature review, since the search’s scope is pre-established. However it is rather inflexible and risks becoming an extremely monumental undertaking. From prior experience attempting such a study, keyword searches return an extreme quantity of results, even when limiting the search by specific date ranges and journal titles. Moreover, the terminology that archaeologists use may not match the terms we use as scholars of scientific practice. Having to account for more terms produces more duplicates and false positives, which must be manually winnowed to a dataset of a viable, yet still realistic size.
Another approach is to examine the contents of all writing produced in a compendium of selected journals or publication venues. This requires justification for why the journals are selected and how this aligns with the study’s scope. This is viable in situations where the study is investigating publication practices (i.e. referencing R scripts, as per Schmidt and Marwick (2020)), but is harder to justify when exploring the scientific content of published work.
4.6 Parsing references from PDFs
It has become clear that the quality of scholarly metadata is very poor for anything that is not an english peer-reviewed STEM article. Through a cursory examination of the major databases (Scopus, Web of Science, OpenAlex, CrossRef, etc), we found that these sources are not useful for our corpus, which includes many monographs and grey literature in languages other than english. It may therefore be better to extract references directly from the text of each work, at least for works that are not in those databases, using a tool like Grobid.
These sources can then be reconciled with the rest of the database through a common citekey format (first author’s last name + year of publication, and then double check all duplicates). Grobid also has a “consolidation” service that normalizes references against the CrossRef (or other) database.
This requires us to have good quality PDFs of each work, which Julie does have through her private library and teaching materials. Moreover, this also presents us with an opportunity to be more thoughtful in assembling our corpus. For instance, we may want to exclude whole books because the disproportionate number of references they contain will surely distort the citation networks (especially when it comes to centrality metrics). So, if a book is actually presenting the culmination of a few articles by the same author, it may be more prudent to include those instead of the book. Basically, this is an opportunity to transform the bibliography from the seed paper — which was never intended to be used in this way — into a functional and purposeful dataset.
So here is a basic protocol for vetting the quality of work that will be put into Grobid. Take detailed notes while doing this vetting. This involves making decisions, and we need to be transparent and reflexive about how we are developing our corpus.
- Filter out any journal articles; assume journal articles from the past 10-15 years have proper scholarly metadata.
- Flag any books that may be better represented by articles by the same authors based on your own experience and familiarity with the community of practice, or which may not be suitable to include in our analysis (i.e. a museum catalog).
- For all books, book chapters, etc:
- Make sure we have the PDF available.
- Prioritize native PDFs, not scanned documents.
- If scanned, then ensure it is good quality.
- Make sure the PDF is of the appropriate page range.
- For book chapters:
- Clip any pages before the first page.
- Clip any pages after the last page of the bibliography.
- If the book assembles all references into a common bibliography, flag the record for manual entry.
- For whole books and/or theses:
- Ensure the PDF contains the full contents, i.e. not just a preview of the front matter.
- For book chapters:
- Make sure the metadata is correct:
- Verify that the item is correctly identified as a book, book chapter or journal articles.
- For book chapters, make sure that the book title, editors, publisher, place, and page range are correct.
- For book chapters, verify that book editors are not listed as authors of the chapters.
- Make sure we have the PDF available.
5 Outcomes
- Venues for publishing and presenting this work
- Workshop
- Public outreach
6 Logistics
6.1 Workflows
The is a quarto-based project situated in a git repository (BibVik). Zack is the repository’s primary maintainer, but Isak has been granted full administrative access, including the ability to make commits directly to the repository without submitting a pull request. Zack and Isak will clearly communicate about what they are working on so as to avoid generating conflictng commit histories which would produce a messy merge scenario.
For less technical users, it would be best to annotate automatically-generated PDFs or to work with an un-tracked copy of the markdown files, whose diffs can then be identified and applied by someone who is more familiar with using git.
Here are a few procedural notes and workflows:
- All writing is done in the file pertaining to the specific section, which are compiled into
BibVik.qmd, which is where the paper’s main parameters are specified. - References are cited using
@followed by the citekey, which follows a strict pattern of the first author’s surname followed by the year of publication, all lowercase and without spaces, e.g.@lund2022. - References are stored in
assets/BibVik.bibwhich is an automatically-updated export of Zack’s Zotero library. If you want to include a reference that is not yet in the library, refer to the DOI and Zack will add it in his revisions. - Although we are keeping this as a private repository for now, all of this can be made into a public website. To preview the site locally, use
quarto previewand navigate tolocalhost:7777in your browser.
Here is an overview of the directory structure:
BibVik/ # Main directory
├── BibVik.qmd # The paper's config file
├── sections/ # Files corresponding with each of the paper's sections
├── 01_introduction.qmd
├── 02_background.qmd
├── etc.qmd
├── assets/ # Files that support quarto rendering
├── BibVik.bib # Biblatex file (currently synced with Zack's entire Zotero library)
├── chicago-author-date.csl # Citation style language used to generate pretty citations
├── title.tex # LaTeX include to generate pretty PDFs
├── data/ # Where the data live
├── figures/ # Where figures live
├── R/ # R functions called from the embedded code blocks
├── research-protocol.qmd # This planning document
├── README # Project documentation
├── _quarto.yml # Quarto config file
├── .gitignore # Git support file
├── .luarc.json # Ignored quarto support file
├── .quarto # Various quarto support files
├── _site # Where the rendered website files live6.2 Roles and responsibilities
Julie Lund: Project lead and point person for anything related to Viking Age archaeology, and the professional community and interests that this paper is concerned with.
Isak Roalkvam: Contributes insights about Viking Age archaeology and the professional community. Works with Zack to implement methodological decisions and processes in code.
Zack Batist: Contributes experience doing bibliometric research in the domain of archaeology. Works with Isak and Julie to derive sound methodological decisions and ensure that they are implemented in a manner that produces valuable and defensible outcomes.
Research assistant: Responsible for qualitatively evaluating the content of research papers that form the basis of this study.
6.3 Funding
The Research Assistant’s salary is funded by the Faculty of Humanities here at the University of Oslo.
6.4 Timeline
TBD
7 References
Footnotes
I wonder if we could also examine, or perhaps publish, an edited version of the peer reviews.↩︎
Note that keywords submitted by authors are not included in this list. This is due to their simplicity and the likelyhood of them being irrelevant to our own objectives. Additionally, some data sources (i.e. anthologies or edited volumes) do not typically use keywords, which will limit our ability to draw comprehensive comparisons.↩︎