Expert identification & selection protocol

Version 1.0.0

1. Rationale

The quality of a Delphi study hinges on recruited experts. This procedure emphasizes (a) a transparent definition of “expert,” (b) a structured, replicable sampling workflow, and (c) diversity across viewpoints to reduce systematic bias. It follows established Delphi guidance on panel composition, size, selection, and documentation, and draws on recent exemplars in psychology and adjacent fields.

2. Panel scope and target size

Target N: ~10 experts (min. 6; max. 16)
Coverage: theoretical psychology, philosophy of science, formal/computational modeling
Rationale: Balances breadth and feasibility; stratification prevents dominance by a single tradition.

3. Step-by-step identification & sampling

Step 1. Seed corpus & name harvesting

Overview of the current state and status of theory: Recently, a statement was created by 26 scientists on the current state and status of theory in psychological science. In this statement, over 50 publications were cited that discuss the current problems with psychological theory and how to improve the situation. Using terms such as theory development, formalization, abduction, explanation, model evaluation, theory education, theory methods, and theory methodology to search through titles, abstracts, and keywords, publications from this list will be identified that could potentially be written by invitable experts.
Topical publications: In the last five years, there have been two special issues on the topic of theory in psychological science (1, 2). Using terms such as theory development, formalization, abduction, explanation, model evaluation, theory education, theory methods, and theory methodology to search through titles, abstracts, and keywords, papers will be identified that could potentially be written by invitable experts.
Literature scan: Additional literature will be found through PsycINFO, Scopus, and Web of Science. Using terms such as theory development, formalization, abduction, explanation, model evaluation, theory education, theory methods, theory methodology, additional articles will be identified. Only articles published in the discipline of psychology, in English, and after 2000 will be considered, and only the first 50 results will be used (ordered by “Relevance”). This will be extended by 50 if insufficient experts can be identified in the initial search.
Identify potential experts: First and last authors from papers on topics identified by the terms used in the literature scan (e.g., methods papers, pedagogy pieces, essays on the need for better theory development).
Rationale: Grounds the panel in demonstrable domain contributions. The papers need to be written by practicing experts (>2000) the experts need to understand each other (English), and the projects concern psychological science (discipline = psychology).

Search Queries

Scopus

A) Theory creation/development/formalization
TITLE-ABS-KEY( (theor* W/3 (develop* OR construct* OR formal* OR build*))
OR “theory building” OR “theory construction” OR “formal theory”
OR (model* W/3 formal*) )
AND (LIMIT-TO (SUBJAREA, “PSYC”)) AND PUBYEAR > 2000

B) Abduction / explanation
TITLE-ABS-KEY( abduction OR “abductive reasoning” OR “inference to the best explanation” OR IBE
OR (explan* W/3 (power OR quality OR adequacy)) )
AND (LIMIT-TO (SUBJAREA, “PSYC”)) AND PUBYEAR > 2000

C) Model evaluation / theory choice / hypothesis derivation
TITLE-ABS-KEY( (model* W/3 (evaluat* OR compar* OR criticism OR adequacy OR selection OR choice))
OR (theor* W/3 (evaluat* OR apprais* OR choice OR compar*))
OR “hypothesis derivation” )
AND (LIMIT-TO (SUBJAREA, “PSYC”)) AND PUBYEAR > 2000

D) Theory education / pedagogy
TITLE-ABS-KEY( (theor* W/3 (education OR pedagogy OR curriculum OR syllabus OR teach* OR training))
OR “theory education” OR “theory pedagogy” )
AND (LIMIT-TO (SUBJAREA, “PSYC”)) AND PUBYEAR > 2000

Web of Science Core Collection

Filters
Research Areas = Psychology;
Document Types = Article, Review article, Book chapter
Timespan = 2000–2025

A) Theory creation/development/formalization
TS=( (theor* NEAR/3 (develop* OR construct* OR formal* OR build*))
OR “theory building” OR “theory construction” OR “formal theory”
OR (model* NEAR/3 formal*) )

B) Abduction / explanation
TS=( abduction OR “abductive reasoning” OR “inference to the best explanation” OR IBE
OR (explan* NEAR/3 (power OR quality OR adequacy)) )

C) Model evaluation / theory choice / hypothesis derivation
TS=( (model* NEAR/3 (evaluat* OR compar* OR criticism OR adequacy OR selection OR choice))
OR (theor* NEAR/3 (evaluat* OR apprais* OR choice OR compar*))
OR “hypothesis derivation” )

D) Theory education / pedagogy
TS=( (theor* NEAR/3 (education OR pedagogy OR curriculum OR syllabus OR teach* OR training))
OR “theory education” OR “theory pedagogy” )

APA PsycInfo (EBSCOhost)

Filters:
Source Types: Academic Journals, eBooks, Literature Reviews
Timespan - 2000 - 2025

A) Theory creation/development/formalization
TX ( theor* N3 (develop* OR construct* OR formal* OR build*)
OR “theory building” OR “theory construction” OR “formal theory”
OR (model* N3 formal*) )
AND TX ( psycholog* )

B) Abduction / explanation
TX ( abduction OR “abductive reasoning” OR “inference to the best explanation” OR IBE
OR explan* N3 (power OR quality OR adequacy) )
AND TX ( psycholog* )

C) Model evaluation / theory choice / hypothesis derivation
TX ( model* N3 (evaluat* OR compar* OR criticism OR adequacy OR selection OR choice)
OR theor* N3 (evaluat* OR apprais* OR choice OR compar*)
OR “hypothesis derivation” )
AND TX ( psycholog* )

D) Theory education / pedagogy
TX ( theor* N3 (education OR pedagogy OR curriculum OR syllabus OR teach* OR training)
OR “theory education” OR “theory pedagogy” )
AND TX ( psycholog* )

Step 3. Exclusion

Identified potential experts will be excluded if they meet one or more of the following criteria.

Their publication predates 2000
The abstract does not explicitly mention that the paper is about, in some shape or form, how to (partly or completely) create, develop, and evaluate scientific theories, or scientific explanations, or about teaching the necessary skills.
They are retired or have the emeritus status
They are early-career researchers (Postdoctoral researcher or lower)

Step 4 — Selection and invitations

Identified potential experts are ordered alphabetically by surname, excluding prefixes (e.g., the Dutch “van”)
Experts are invited in descending order until the quota is met.
To protect against a men-only expert group, at least ⅓ of the invitees need to be non-male.
Over-recruit by ~30% to offset attrition. The remaining experts are maintained as a waitlist.

Step 5. Bounded snowballing

Experts identified and invited through Step 1 are asked to identify additional experts
These seed experts are asked to nominate up to 3 additional experts (especially early-career or underrepresented scholars).
Include nominees only if they meet the inclusion criteria.
Rationale: Broadens coverage while guarding against reputational bias.