McCann Associates

Job Analysis

Job analysis is a fundamental part of all well-developed assessment programs, and provides the objective, legally defensible criteria needed to support your assessment program. Job analysis is the systematic study of a job that documents the knowledge skills, and abilities necessary for successful performance of the job under evaluation. McCann has over 50 years of experience designing and developing job analysis survey instruments, collecting and analyzing job analysis data, and preparing job analysis reports. Our various methods of collecting data range from surveys, to focus groups, to data review (training materials, job descriptions, rules and regulations, etc.), to on-the-job observations to in-person and/or telephone interviews. In addition, because the knowledge, skills and abilities required to perform a specific job may change overtime, we advise refreshing your job analysis every five (5) years.

Test Specification Development

The test specification, sometimes referred to as the "test blueprint," defines the structure of the test. Using the results of the job analysis, the "test blueprint" defines the content areas and appropriate weight of each conent area, the number and type of items on the test for each content area, and may also includes scoring and reporting procedures. Our methods of test specification development fortify content validity by relying on job analysis data and consultations with subject matter experts (SMEs).

Item Writing Workshops

Our Item Writing Workshops are designed to help your organization create valid items aligned to your program's test objectives, test blueprint, and job analysis. We will assist you in developing a test blueprint to ensure content validity and, with our expert test development staff, your new items will meet or exceed industry item development standards. In addition to our face-to-face workshop services, our psychometrically trained staff are experienced in conducting virtual item writing workshops, which eliminate the time and financial expense associated with travel and accommodations for your item writers. Our workshops begin with a training session, during which our test development experts orient the item writers to the characteristics of valid test items, particularly emphasizing alignment to your test blueprint to ensure content validity. Training includes a thorough review on how to write grammatically correct, clear items that conform to industry test standards, as well as the importance to reference each item to the blueprint and job analysis (or other material as deemed relevant). Together, the group reviews a training set of items and discusses whether the items meet industry standards and whether they exhibit all the characteristics of a "good" item. Item writers then practice writing a few items, which are then critiqued by the group. Once training ends, based on each item writer's area of expertise, item writing assignments are distributed. Face-to face item writing workshops generally last 3 to 5 days, while our virtual item writing workshops accommodate essentially any schedule. Once the new pool of items are generated, each item is psychometrically reviewed by our team of item writing experts.

Benefits:

• Generate a pool of items that are content valid, aligned with your test blueprint, meeting industry item writing standards, ready for piloting
• Access to secure, online, virtual item writing tools to empower your item writers to perform tasks at home, thereby eradicating substantial travel and business expenses incurred with face-to-face item writing workshops
• Create items directly into the online item bank, thereby removing back-end work of re-entering items into a new system
• Assign work flow tasks such as item writers, item reviewers and item approvers
• Easily generate item bank reports to find gaps, item status (draft, in review, approved), and other pertinent item related data

Test Review

The purpose of a test review is to examine the content covered by an existing test and to determine if the extent to which it is aligned with the test blueprint. During the workshop, subject matter experts (SMEs) are trained in test construction guidelines and work through the process of matching each test item back to the test blueprint.

Item Analysis Review

Item analysis review workshops are conducted to ensure all items on a test are performing in appropriate ways which contribute to the overall reliability and validity of a test. Prior to conducting the workshop, our psychometric staff uses a variety of statistical methods to describe item behaviors, detect items that may be adversely affecting the psychometric properties of the overall test, or are otherwise performing in unintended ways that can challenge the reliability and validity of the test for specific demographic subgroups.

These methodologies include classical test theory (CTT) and item response theory (IRT) measures of item difficulty, item discrimination, as well as measures of differential item functioning (DIF), such as: Mantel-Haenszel, logistic regression, and/or item response theory (IRT) indices. Based on the results of these analyses, our staff flags any item that falls below standard acceptable performance levels.

Adverse impact analyses, such as: the 80% rule, Fisher's Exact Test and the Binomial Test, which provide information on the test-level performance of demographic subgroups may also be included in the item analysis review.

During the workshop, subject matter experts (SMEs) are trained on statistical concepts relating to determining the validity, reliability of test items and test and item bias. With our psychometric staff, the panel of SMEs reviews each flagged item to determine the most appropriate next steps to protect both the fairness and content validity of the test.

Essay and Constructed-response Assessment Modeling

IntelliMetric®, our automated essay scoring system is the "gold-standard" in automated essay scoring. Nearly ten years of research studies show that Intellimetric equals or exceeds the accuracy of expert human scorers, and is appropriate for both low and high stakes assessment environments, and virtually any type of written content relevant to our clients' written assessment needs. Intellimetric's artificial intelligence analyzes more than 300 semantic, syntactic, and discourse characteristics and provides a holistic score as well as scores within five major domains: focus and meaning, organization, content and development, language use and style, and mechanics and conventions. Our Intellimetric tool is web-based, available anywhere at any time, is less expensive than traditional offline administration and hand-scoring, and has scoring capabilities in over 20 different languages.

Oral Assessments

McCann's psychometrics and test development team has over 50 years of experience working with clients to develop and deliver structured oral interview assessments that measure the knowledge, skills, and abilities (KSAs) necessary for successful performance in managerial and supervisory positions. The content of our promotional oral exams focus on critical incidents encountered on the job and are developed in conjunction with subject matter experts (SMEs). Members of our oral board panels are experienced SMEs, and are well-trained in objectively and impartially applying promotional oral assessment rating scales.

Benefits:

• Superior content and predictive validity and reliability for knowledge, skills, and abilities that are integral to successful managerial and supervisory job performance
• Procedures that are fair to EEOC protected groups, such as minorities and women
• Expert evaluators that have years of supervisory experience
• 50 years of experience in designing, developing and administering structured oral interviews

Assessment Centers

Assessment centers are designed to assess skills and abilities that paper/pencil tests simply can't measure as accurately. Assessment centers are ideal in some limited circumstances, for example, to measure critical thinking, decision-making, problem-solving, and interpersonal skills. Assessment centers provide standardized and objective ways to evaluate candidates on these and other knowledges, skills, and abilities, using realistic scenarios or job simulations. Assessment center exercises can include: role playing, leaderless discussions, fact-finding and in-basket exercises. Candidates participate in these activities, and are rated on their performance by a panel of highly trained subject matter experts (SMEs).