撰写同侪评审指南

同侪评审指南：绩效评估与360度反馈

一、目的与适用范围

• 目的：为评审者提供一套基于证据、可操作且具有心理测量学与组织应用依据的评审标准与流程，用于审查组织内的绩效评估与360度反馈（下称“360反馈”）项目的设计质量、实施合规性与使用效度。
• 适用范围：适用于用于发展性与/或决策性（如晋升、加薪）用途的绩效评估与360反馈项目，包括工具开发、样本与实施、评分与汇总、信效度与公平性证据、数据治理与伦理、反馈交付与后续应用等。

二、核心原则与评审角色规范

• 科学性与证据导向：评审意见须以工作分析、心理测量学证据与实证研究为依据（AERA, APA, & NCME, 2014；DeNisi & Murphy, 2017）。
• 公平与合规：确保过程公平、结果可解释、隐私受保护，符合适用标准与法规（ISO 10667-2:2020；UGESP, 1978）。
• 适配与用途明确：区分发展性与决策性用途；用途决定证据门槛与报告策略（Bracken, Rose, & Church, 2016）。
• 评审独立性：回避利益冲突；保持材料保密；记录可追溯的判断依据与建议。

三、评审维度、判据与证据要求 A. 目标一致性与岗位分析

• 判据：项目目标（发展/决策/混合）明确；与战略与岗位能力模型一致；基于最新、结构化岗位分析（任务-能力映射、关键事件法等）。
• 证据：岗位分析报告；能力词典与行为指标；用途声明与政策。
• 关键点：避免抽象人格形容词，采用可观察的行为指标（Smith & Kendall, 1963）。

B. 工具与内容质量（量表与条目）

• 判据：条目与能力维度一一对应；行为锚定清晰；量表有“无法观察/不适用”选项；语言简洁、文化与语境适切；避免双重负载条目。
• 证据：条目蓝本与认知访谈记录；专家内容效度评审；试测与修订记录。
• 建议：优先采用行为锚定评分量表（BARS）或行为频率量表，以降低晦涩性与晕轮效应（Smith & Kendall, 1963）。

C. 评价者取样与实施

• 判据：评价者来源覆盖（上级/同事/下属/自评/客户）与数量满足匿名性与信度需要；实施流程标准化；评价者培训充分。
• 证据：取样框架与最小组容量规则；培训材料与到课率；提醒与申诉机制。
• 基线要求：每一评价来源建议≥3人以保护匿名；≥5–7人可提升可靠性（Bracken, Timmreck, & Church, 2001）。对高风险决策用途，优先更大样本并提供替代来源（例如客户）。
• 培训要点：框架参照（frame-of-reference）训练、行为观察与证据记录、偏差认知（宽厚/严苛、中心趋向、晕轮、近因）、保密承诺（DeNisi & Murphy, 2017）。

D. 评分、汇总与统计方法

• 判据：评分规则、缺失与异常处理、跨来源加权与汇总事先预注册并与用途一致；统计模型可解释、可复核。
• 证据：评分与加权方案（含公式）；缺失值与极端反应处理规则；报告模板与示例。
• 建议：
  • 缺失：将“无法观察”视作缺失；避免将缺失计为低分。
  • 汇总：报告分来源结果与总体综合分；决策用途时预先定义来源权重（例如上级权重较高）并给出依据（岗位任务分析）。
  • 质量控制：剔除低质量反应（极短作答时长、直线作答、同质极端）需有阈值与记录。
  • 高阶方法：对高风险用途建议进行多面向Rasch或广义化理论分析以分离项目难度、评价者严厉度与被评价者真实差异（Myford & Wolfe, 2003；Shavelson & Webb, 1991）。

E. 信度与效度证据

• 判据：提供针对目标人群与用途的信度与效度证据，分来源报告。
• 证据：
  • 信度：内部一致性（α/ω），跨评价者一致性（ICC；建议报告ICC(2,k)）、一般化理论G研究与D研究（Shavelson & Webb, 1991）。
  • 结构效度：维度性检验（CFA/多特质-多方法MTMM以识别来源方法效应）。
  • 关联效度：与客观绩效、业务KPI、上级评估的关系；已知群体效应与收敛-区分效度。
  • 后果效度：干预后行为改变与绩效改进证据（Kluger & DeNisi, 1996；London & Smither, 1995）。
• 判定门槛：发展性用途可接受中等信度；用于人事决策时应达到较高水平，且多重证据一致（AERA et al., 2014）。

F. 公平性与偏差控制

• 判据：识别并控制来源方法效应与评价者特质偏差（如“评价者特异效应”）；开展差异功能（DIF）与测量不变性检验；监测不良影响。
• 证据：方差分解（人×来源×项目）；性别/年龄/族群/岗位等级不变性测试；不良影响与解释。
• 要点：评价者特异效应在评分方差中占比可观，需通过多来源汇总、评价者训练与模型校正降低影响（Scullen, Mount, & Goff, 2000）。遵循公平性与可及性准则（AERA et al., 2014）。

G. 数据治理、隐私与伦理

• 判据：合规的告知同意、数据最小化、访问控制、匿名/去标识、保留与删除策略；报告设置最小展现门槛（如每来源≥3份）。
• 证据：隐私政策、数据流程图、访问角色矩阵、安全审计记录；与ISO 10667或等效标准对齐的服务流程。
• 要点：分离发展性报告与管理者可见信息，防止报复与寒蝉效应（Bracken et al., 2016）。

H. 反馈报告与应用

• 判据：报告清晰、可解释、可行动；辅以教练或复盘；目标设定与跟进机制完善。
• 证据：报告样例（含行为证据与具体建议）、教练与行动计划模板、跟踪节点与指标。
• 要点：
  • 附带规范化参照（同岗/同级标杆）与信度/不确定度提示。
  • 发展性用途优先；将360直接用于薪酬或晋升的项目需提供对评价诚实性与效度影响的缓释机制（如匿名保护、独立第三方实施）（Atwater & Brett, 2005；Bracken et al., 2016）。
  • 将反馈转化为具体目标与实施计划可显著提升效果（Locke & Latham, 2002）。

I. 持续改进与监控

• 判据：设定关键质量指标（完成率、项目方差、ICC、有效评论率、申诉率等），定期审计并修订工具。
• 证据：周期性质量报告、版本更新记录、变更影响评估。

四、评审流程与输出

• 准备阶段：收集材料清单（见附录）；明确用途与情境；确认利益冲突与保密协议。
• 独立评审：依据上述九大维度形成分项判定与证据点评。
• 合议与定稿：若多名评审者，先独立打分后开展校准会议；形成一致意见与优先级排序的整改建议。
• 输出格式：每一维度给出等级判定与关键证据摘要，并提供具体、可操作的改进建议与时间表。

五、建议的分级判定与释义

• 不充分：关键证据缺失或存在重大风险（如无岗位分析、匿名门槛不足、信度不达标）。
• 基本充分：核心流程具备但证据薄弱或不一致（如仅单一来源信度合格）。
• 充分：证据链完整、指标达标、风险可控（含公平性与隐私保护）。
• 卓越：除充分外，采用先进方法（如多面向模型）、系统化教练与持续改进闭环，且有实证的组织绩效改进证据。

六、常见风险与纠偏建议

• 将360反馈直接用于薪酬决策导致评价失真与反馈接受度下降：先以发展性用途试点，强化匿名保护与第三方实施，并分阶段引入管理用途（Bracken et al., 2016）。
• 条目过于抽象或道德化：改写为具体、可观察行为，提供积极/消极锚点示例（Smith & Kendall, 1963）。
• 评价者样本不足或结构失衡：扩充同级与跨职能评价者，确保每来源≥3且尽量达到5–7人（Bracken et al., 2001）。
• 统计处理不透明：预注册汇总与缺失处理规则，提供方法附录与可复核的计算说明。
• 方法效应与评价者特异效应偏高：采用MTMM/广义化理论/多面向Rasch分离偏差；开展框架参照训练（Myford & Wolfe, 2003；Shavelson & Webb, 1991）。
• 反馈难以转化为改进行动：配置教练会谈、目标设定与里程碑检查；强调“前馈”与具体化（Kluger & DeNisi, 1996；Locke & Latham, 2002）。

七、评审材料清单（提交方应提供）

• 岗位分析报告与能力模型
• 工具蓝本（条目池、量表设计、行为锚定）
• 专家评审与试测报告（含修订记录）
• 实施规范（取样策略、匿名与最小单元规则、培训材料）
• 评分与汇总规则（缺失/异常处理、加权方案、预注册文档）
• 信度与效度报告（分来源；含ICC、结构模型、关联效度）
• 公平性分析（不变性/DIF/不良影响监测）
• 数据治理与隐私合规文件（ISO 10667/等效对齐说明）
• 报告与反馈交付样例、教练与行动计划模板
• 周期性质量监控报表与改进记录

参考文献（APA第7版）

• AERA, APA, & NCME. (2014). Standards for educational and psychological testing. American Educational Research Association.
• Atwater, L. E., & Brett, J. F. (2005). Antecedents and consequences of reactions to developmental 360 feedback. Journal of Occupational and Organizational Psychology, 78(3), 473–490.
• Bracken, D. W., Rose, D. S., & Church, A. H. (2016). The evolution and devolution of 360-degree feedback. Industrial and Organizational Psychology, 9(4), 761–794.
• Bracken, D. W., Timmreck, C. W., & Church, A. H. (Eds.). (2001). The handbook of multisource feedback. Jossey-Bass.
• DeNisi, A. S., & Murphy, K. R. (2017). Performance appraisal and performance management: 100 years of progress? Journal of Applied Psychology, 102(3), 421–433.
• Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254–284.
• London, M., & Smither, J. W. (1995). Can multi-source feedback change perceptions of goal accomplishment, self-evaluations, and performance-related outcomes? Personnel Psychology, 48(4), 803–839.
• Myford, C. M., & Wolfe, E. W. (2003). Detecting and measuring rater effects using many-facet Rasch measurement. Journal of Applied Measurement, 4(4), 386–422.
• Scullen, S. E., Mount, M. K., & Goff, M. (2000). Understanding the latent structure of job performance ratings. Journal of Applied Psychology, 85(6), 956–970.
• Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Sage.
• Smith, P. C., & Kendall, L. M. (1963). Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales. Journal of Applied Psychology, 47(2), 149–155.
• ISO. (2020). ISO 10667-2:2020 Assessment service delivery—Procedures and methods to assess people in work and organizational settings—Part 2: Requirements for service providers.
• Uniform Guidelines on Employee Selection Procedures, 43 Fed. Reg. 38290 (1978).
• Locke, E. A., & Latham, G. P. (2002). Building a practically useful theory of goal setting and task motivation. American Psychologist, 57(9), 705–717.

使用说明

• 评审时建议将上述九大维度作为主目录，逐条记录证据与判定，并在结论部分明确总体风险等级与优先改进项（不超过5条），同时标注对项目用途（发展性/决策性）的可行性判定与必要前置条件。

Peer Review Guidelines for Manuscript Evaluation

Purpose and scope

• These guidelines standardize high-quality, ethical, and evidence-based peer review of scholarly manuscripts across disciplines. They specify reviewer responsibilities, core evaluation criteria, study-type–specific expectations, reporting standards, and structured reporting formats, aligned with recognized ethical and methodological authorities (e.g., COPE; ICMJE; EQUATOR Network).

1. Ethical principles and reviewer responsibilities

• Confidentiality and data protection: Treat all materials as confidential; do not share, store insecurely, or use for personal research. Do not upload manuscripts or data to third-party tools without explicit journal permission (COPE, 2017; COPE, 2023).
• Competence and scope: Accept reviews only when the manuscript aligns with your expertise and you can meet the deadline; otherwise decline promptly and, if requested, suggest qualified alternatives (COPE, 2017).
• Conflicts of interest: Disclose any financial, professional, personal, or intellectual conflicts that could bias judgment (ICMJE, n.d.; COPE, 2017).
• Impartiality and fairness: Evaluate solely on scholarly merit, avoiding favoritism, hostility, or identity-based bias; adhere to double-anonymous processes when applicable.
• Research integrity vigilance: Flag potential plagiarism, duplicate submission, image manipulation, data fabrication, undeclared conflicts, or ethical issues to the editor confidentially with objective evidence (COPE, 2017).
• Responsible use of generative AI: Do not rely on AI tools to read confidential manuscripts or to generate review content unless explicitly permitted by the journal; you remain accountable for all content (COPE, 2023).
• Timeliness and diligence: Deliver thorough reviews on time; if delays arise, notify the editor promptly.

2. Pre-review readiness checklist

• Fit: The manuscript aligns with your methodological and topical expertise.
• Feasibility: You can review thoroughly within the timeline.
• Independence: No conflicts of interest exist or they are fully disclosed and accepted by the editor.
• Standards familiarity: You can apply relevant reporting guidelines (EQUATOR Network), statistical and methodological standards (e.g., CONSORT, PRISMA 2020, STROBE, STARD, SRQR/COREQ; ASA), and transparency norms (TOP Guidelines; FAIR data).

3. Core evaluation criteria and rating rubric Use a 5-point scale for each criterion and provide evidence-based justifications and actionable suggestions.

• Originality and contribution 1 = Trivial or duplicative; 3 = Modest incremental advance; 5 = Clear, novel contribution that shifts understanding or practice.
• Significance and relevance 1 = Low scholarly or practical value; 3 = Moderate importance; 5 = High theoretical, methodological, or practical impact for the field and journal audience.
• Theoretical framing and literature integration 1 = Weak or outdated; 3 = Adequate with gaps; 5 = Robust, current, and logically linked to aims and hypotheses.
• Methodological rigor and design appropriateness 1 = Major flaws or misalignment; 3 = Generally sound with notable limitations; 5 = Design is appropriate, justified, and rigorous for the research question.
• Measurement quality and construct validity 1 = Poorly defined constructs, unreliable measures; 3 = Adequate validity/reliability; 5 = Strong evidence for validity, reliability, and appropriate operationalization.
• Data analysis and statistical reporting 1 = Incorrect or opaque analyses; 3 = Mostly appropriate with improvements needed; 5 = Correct, transparent, with effect sizes, uncertainty, and assumptions addressed (ASA; SAMPL; APA JARS).
• Transparency, reproducibility, and openness 1 = Insufficient detail; 3 = Partial transparency; 5 = Protocol/preregistration as applicable, data/code/materials availability or justified restrictions, adherence to TOP and FAIR principles.
• Ethical and governance compliance 1 = Unclear or noncompliant; 3 = Partially documented; 5 = Ethics approval/consent documented, risks mitigated, equity considered, data privacy managed.
• Interpretation and limitations 1 = Overstated or unsupported claims; 3 = Mixed; 5 = Conclusions warranted by evidence, limitations and generalizability transparently discussed, appropriate policy/practice implications.
• Reporting quality and structure 1 = Unclear, noncompliant with reporting guidelines; 3 = Adequate with gaps; 5 = Clear, coherent, guideline-compliant, replicable descriptions.
• Scholarly apparatus (citations and positioning) 1 = Incomplete/biased; 3 = Adequate; 5 = Comprehensive, current, balanced use of primary sources.
• Equity, diversity, inclusion, and accessibility 1 = Unaddressed with potential harm; 3 = Partially considered; 5 = Thoughtfully integrated in design, sampling, measures, analysis, and reporting.

Suggested weighting (modifiable per journal): Contribution/significance (20%), Methods (20%), Analysis/reporting (20%), Transparency/ethics (15%), Interpretation (15%), Writing/scholarship/EDI (10%).

4. Study-type–specific expectations and checklists Apply general criteria above plus the following study-specific standards.

• Randomized trials and experimental studies

  • Adhere to CONSORT 2010 and extensions (e.g., cluster, noninferiority).
  • Verify randomization sequence generation, allocation concealment, blinding, pre-specified outcomes, sample size calculation, protocol registration, ITT analyses, harms reporting.
  • Assess multiplicity control, missing data handling, and deviations from protocol.

• Observational studies (cohort, case-control, cross-sectional)

  • Apply STROBE.
  • Evaluate exposure/outcome ascertainment, confounding control (design and analysis), measurement bias, missing data, sensitivity/robustness analyses, temporality, and causal claims discipline.

• Diagnostic/prognostic accuracy

  • Apply STARD 2015.
  • Assess spectrum of participants, index test and reference standard definitions, blinding, thresholds, calibration and discrimination metrics, clinical utility.

• Systematic reviews and meta-analyses

  • Apply PRISMA 2020; check protocol registration (e.g., PROSPERO), comprehensive search, dual screening/data extraction, risk-of-bias assessment (e.g., RoB 2; ROBINS-I), synthesis methods, heterogeneity and small-study bias, certainty-of-evidence frameworks (e.g., GRADE), and transparent exclusions.

• Qualitative research

  • Apply SRQR or COREQ as appropriate.
  • Evaluate sampling strategy and rationale, data collection procedures, reflexivity, analytic approach (e.g., thematic analysis, grounded theory) with auditability, trustworthiness (credibility, dependability, transferability), saturation/adequacy.

• Survey research

  • Apply AAPOR transparency standards and, where applicable, APA JARS–Quant.
  • Assess sampling frame, mode and coverage, response/retention rates with standardized definitions, weighting and variance estimation, questionnaire design and pretesting, measurement validity and reliability, nonresponse bias assessment.

• Measurement development and validation

  • Evaluate content validity evidence, dimensionality (EFA/CFA), reliability (omega preferred over alpha when appropriate), measurement invariance, item response theory as appropriate, cross-validation and sample size adequacy, scoring, and interpretability.

• Computational/ML modeling and prediction

  • Assess data provenance, preprocessing, leakage risks, internal/external validation, performance metrics (discrimination, calibration), uncertainty quantification, fairness/equity assessment, robustness/stress testing, complete code and hyperparameter transparency.

5. Statistical and analytical reporting expectations

• Report effect sizes with confidence intervals; avoid sole reliance on null-hypothesis significance testing; interpret p-values in context (Wasserstein & Lazar, 2016; Wasserstein, Schirm, & Lazar, 2019).
• Justify model choices; assess assumptions; describe missing data patterns and handling; pre-specify primary analyses where appropriate; address multiplicity.
• Provide analysis code and data dictionaries whenever ethically and legally possible; justify any restrictions (TOP; FAIR; SAMPL; APA JARS).

6. Structuring the review report

• Summary (2–5 sentences): State the study’s aims, methods, main findings, and contribution in neutral terms.
• Major strengths (bulleted): Identify substantive merits (e.g., novelty, rigorous design).
• Major concerns (numbered): For each, (a) state the issue; (b) explain why it matters with reference to standards or evidence; (c) provide concrete, feasible recommendations (e.g., additional analyses, clarifications, revised claims).
• Minor issues (bulleted): Clarity, formatting, small methodological clarifications; avoid line editing unless clarity impedes interpretation.
• Confidential comments to the editor: Suitability for the journal, ethical concerns, overlap with prior work, reviewer confidence and remaining uncertainties.
• Ratings and recommendation: Provide criterion ratings and an overall recommendation with rationale.

7. Decision recommendations and thresholds

• Accept: All criteria strong; only typographic or minimal clarifications needed.
• Minor revision: Core methods sound; limited, addressable concerns not affecting conclusions materially.
• Major revision: Substantial issues in design, analysis, or reporting that could be addressed; conclusions may change.
• Reject: Fundamental flaws (e.g., invalid design for the question, irreparable bias), inadequate contribution, or ethical noncompliance. Provide constructive guidance for potential resubmission elsewhere.

8. Bias mitigation and professional conduct

• Reflect on sources of bias (e.g., topic preferences, institutional reputations, methodological schools).
• Use evidence-based critiques, cite standards (e.g., CONSORT, PRISMA 2020), and avoid prescriptive personal preferences absent justification.
• Maintain a respectful, specific, and solution-focused tone. Do not reveal your identity unless journal policy permits.

9. Common red flags necessitating editorial attention

• Ethical concerns: Lack of IRB/IEC approval where required, inadequate consent, data privacy risks.
• Integrity concerns: Plagiarism, image/data manipulation, impossible results or reused datasets without disclosure.
• Salami slicing or duplicate publication.
• Undisclosed conflicts of interest or funding influence.

10. Reviewer checklist (abbreviated)

• Ethical compliance documented (approval/consent/data use).
• Adherence to appropriate reporting guideline(s) confirmed.
• Design and measures fit the research questions.
• Analyses correct, transparent, and reproducible; effect sizes/CIs reported.
• Claims proportionate to evidence; limitations and generalizability addressed.
• Data/materials/code availability aligned with journal policy or justified.
• Equity and bias considerations addressed (sampling, measures, outcomes).

References (selected)

• American Association for Public Opinion Research. (n.d.). Transparency and standard definitions resources. https://www.aapor.org
• Appelbaum, M., Cooper, H., Kline, R. B., et al. (2018). Journal article reporting standards for quantitative research in psychology. American Psychologist, 73(1), 3–25.
• Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., et al. (2015). STARD 2015: An updated list of essential items for reporting diagnostic accuracy studies. BMJ, 351, h5527.
• Committee on Publication Ethics. (2017). Ethical guidelines for peer reviewers. https://publicationethics.org
• Committee on Publication Ethics. (2023). Position statement on the use of AI tools in research and publication. https://publicationethics.org
• EQUATOR Network. (n.d.). Enhancing the quality and transparency of health research. https://www.equator-network.org
• International Committee of Medical Journal Editors. (n.d.). Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals. http://www.icmje.org
• Lang, T. A., & Altman, D. G. (2013). The SAMPL guidelines for statistical reporting. In P. Smart, H. Maisonneuve, & A. Polderman (Eds.), Science Editors’ Handbook. European Association of Science Editors.
• Nosek, B. A., Alter, G., Banks, G. C., et al. (2015). Promoting an open research culture. Science, 348(6242), 1422–1425.
• O’Brien, B. C., Harris, I. B., Beckman, T. J., Reed, D. A., & Cook, D. A. (2014). Standards for reporting qualitative research. Academic Medicine, 89(9), 1245–1251.
• Page, M. J., McKenzie, J. E., Bossuyt, P. M., et al. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, n71.
• Schulz, K. F., Altman, D. G., & Moher, D.; CONSORT Group. (2010). CONSORT 2010 statement: Updated guidelines for reporting parallel group randomized trials. BMJ, 340, c332.
• Tong, A., Sainsbury, P., & Craig, J. (2007). Consolidated criteria for reporting qualitative research (COREQ). International Journal for Quality in Health Care, 19(6), 349–357.
• von Elm, E., Altman, D. G., Egger, M., et al. (2007). The STROBE statement. PLoS Medicine, 4(10), e296.
• Wasserstein, R. L., & Lazar, N. A. (2016). The ASA’s statement on p-values. The American Statistician, 70(2), 129–133.
• Wasserstein, R. L., Schirm, A. L., & Lazar, N. A. (2019). Moving to a world beyond “p < 0.05”. The American Statistician, 73(sup1), 1–19.
• Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., et al. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data, 3, 160018.

Note on citation style

• Adapt citation style to the journal’s specified format (e.g., APA, Vancouver). Where study-type–specific guidelines are cited (e.g., CONSORT, PRISMA), include the canonical statement and any relevant extensions in the reference list per the journal’s instructions.