October 2008 Newsletter

Technology has such a profound impact on our every day lives. Who would have thought thirty-years ago that people would be carrying one-by-three inch devices everywhere they go and that they could call anyone in the world with it? If that's not all, who could have imagined that this small electronic device could also tap into the internet, receive play-by-play sports feeds, play any song by any artist, stream movies, record videos, snap pictures, ramp up a GPS system and empower kids to play video games and write text messages? And why do we still call these devices phones? I suppose it's a blend of history, dreams and technological innovations that advanced the communications industry from the rotary to the cellular phone. Hard to believe that children born today may have no recollection of a rotary phone or even a phone booth for that matter, other than reference to such in old movies, history books and Superman. Who knows, maybe even the cell phone as we know it today will evolve in accord with someone's dream and more advanced technology to a point that even it, as it exists right now, becomes just a faded memory in twenty years.

Just as the cell phone has radically changed the way we communicate on almost every imaginable front, McCann Associates blends history, dreams and technological advancements to sharply and positively impact the way organizations train, develop, test, measure, and remediate individuals across multiple disciplines. Today, McCann makes yesterday's dreams a reality. Our acclaimed software solutions empower clients to score essays and constructed response items immediately, reliably and validly (in almost every conceivable language); to conduct virtual online item writing workshops, item edit workshops and item review workshops; to completely control and manage item banks, securely, 24/7 from anywhere; to expand test center channels to include proctored corporate testing, hospital facilities, hotels, and universities (just to name a few); to conduct online job analysis; to deliver assessments in linear, adaptive, branching, and testlet modes; to receive on-the-spot score reports and monitor tests in progress, not started and finished; to deliver video based and other advanced item types; to deliver online training programs and materials linked to assessments and so much more. We implement solutions that, quite frankly, not too long ago were unimaginable.

In addition to shaping the testing and training communities with easy-to-use online solutions, McCann remains steadfast in our commitment to provide paper and pencil solutions to hundreds of clients that prefer, for good, valid and various reasons, to deliver assessments via traditional methods. Our fifty years of experience begins with paper and pencil test delivery and scoring services; and, we will continue to extend these services far into the future.

This month's "Fun, Featured Fact":

Upward bound together, driving your business higher

Many nonprofit organizations operate on razor thin budgets, while contemporaneously facing pressures to provide more services and, plain and simple, just do more, a lot more. Aside from our many online solutions that impart ways to vastly cut current expenses, one of our more popular revenue generating offerings is online practice tests. Practice tests enable organizations to do more by providing an added benefit and service to their constituents, while concurrently reap the rewards of a profitable new revenue stream. We provide everything an organization needs to create and distribute practice tests to members worldwide. And focused, dedicated, serious organizations can launch tests within hours of receiving our services.

In our experience, we find that many organizations have a bank of retired items. Many of these retired items are valid, discriminate well between able and less able candidates, and are aligned to test blueprints and job analyses. While these items are "good" by all psychometric accounts, they are "retired" mainly due to exposure reasons. These items are a great source for a practice test. Practice tests reduce test anxiety because they allow your candidates to get a realistic "feel" for the high-stakes exam. Often they mirror the content, number of items, type of items, and the time limits imposed by the actual test. With our tools, practice tests are scored immediately - even essays - and you control the amount of feedback to deliver to the candidates. Immediate, detailed feedback helps the candidate understand, precisely, areas of strengths and weaknesses and where they need to focus their studies.

We have partnered with several organizations to deliver their practice tests including GMAC (GMAT Write1 and Write2), Cedar Bluff (suite of Physician Assistant Exams), American Board for Certification of Teacher Excellence (PTK and ELA Writing tests), Association of Energy Engineers (Certified Energy Manager Test) and more. I encourage you to consider how we can help drive your business higher, through practice tests or any of our other services.

Pam Kirby

Vice President
Write to me directly at pkirby@mccanntesting.com

The World Is Flat! And the Passport Language Program Provides Educators with Just the Right Type of Information Necessary to Understand Student Language Proficiencies and Gaps

The modern world requires a level of communication previously left to the domain of diplomats and world travelers. In the words of Thomas Friedman, author of The World is Flat "In this era of mounting complexity with more people, systems and products entwined in a bewildering web of global networks, explaining is an enormously valuable skill." Compounding this is the growth in immigration, cross-border education, globalization and regulation. As technology and growing economies around the world have brought disparate people together the need to communicate, control and share knowledge has grown exponentially. This drives a need to better inform educators on student linguistic skills and growth towards the next, higher-level of understanding.

The Passport Language Program (PLP) offers educators a comprehensive method for assessing the language skills of English Language Learners. Aligned to the proficiency levels detailed in the Common European Framework of Reference (CEFR), PLP is a robust application that provides online testing, scoring, reporting, and remediation.

The Passport Language Program's (PLP) B1 Entrance and Certification tests, which measure English Language Learner proficiency for the high school and collegiate levels, have been recently validated on a diverse national sample of high school and college-aged students. The validation results indicate that the PLP entrance and certification tests accurately measure English language skills for English language learners of a variety of first-language backgrounds, including: Chinese, Korean, Spanish, Hebrew, Hungarian, Polish, Telugu, and Turkish. Validation studies for pre-high school and college proficiency levels are underway.

Powered using the Measured Success™ platform, PLP provides an online alternative to paper and pencil tests, which can be time consuming and tedious to score. Measured Success provides the power and functionality that enables the Passport application to provide online testing, scoring, reporting and remediation. The platform can also utilize paper and pencil scoring via our VanScan technology. Using PLP, scores are provided instantly, even on essays, which are scored using the world's best automated essay-scoring engine, Intellimetric™. The application achieves levels of scoring accuracy that equal or exceed expert graders.

CEFR describes achievements for learners of foreign languages across cultural and sociopolitical boundaries. Its objective is to provide a method of assessing and teaching which applies to all languages. It was developed through a process of scientific research and wide consultation and provides a practical tool for setting clear standards to be attained at successive stages of learning and for evaluating outcomes in an internationally comparable manner.

The six reference levels (defined below) have become widely accepted as the standard for grading an individual's language proficiency, including listening, reading, and writing. These levels are broken out into separate bands relative to grade level as follows:

  • Middle School English: Developed to assess students aged 12 to 14 (Level C1)
  • High School English: Developed to assess students aged 14 and older (Levels B1 and B2)
  • Elementary School English: Developed to assess elementary school students under age 12 (Levels A1 and A2)
Contact Us to learn more on implementing PLP in your University.

Public Safety News

It has been our pleasure to work with public safety organizations around the country for over fifty-years. Police officers and firefighters dedicate their lives to protecting our communities, to responding to emergencies, to reducing crimes, and to making our world a safer place to live in. This is, by no means, getting any easier with the ever increasing dangers facing our communities, large and small. Aside from the many crucial traditional services provided by our public safety officers, we now also see huge efforts on their part in connection with crime prevention education, keeping our schools safe, drug prevention educational seminars and a focus on safe driving, just to name a few.

McCann is committed to a continued, dedicated focus on assisting municipalities in hiring and promoting qualified public safety officials. Responding to the mounting pressures placed on public safety officials that translate into additional needs voiced by our clients, we have implemented many changes and new offerings on our end. By way of example, we now offer practice tests for entry level and promotional ranks to help prepare candidates for their high-stakes written tests. All of our high-stakes entry level and promotional written tests are now available online and/or paper and pencil, giving municipalities the option to deliver web-based tests with immediate scoring and reporting. We have expanded our assessment center activities to include not only structured oral interviews, but also in-basket assessments, leaderless discussions, role playing and fact-finding activities. We're working with groups to update their city-specific job analyses to ensure they are delivering valid, reliable tests. Our consulting services have expanded to include test proctoring, open-house seminars, pre and post test reviews, improved appeals processing, adverse impact analysis and more. We are in the midst of collecting nationwide data to re-validate our entry level exams and to determine whether and how to update our current bibliographies. We are exploring the integration of our writing tools into our current suite of public safety offering to assist officers with report writing and to expand testing options to include essays. And finally, we have recently released a turn-key Human Resources module that allows administrators to track, train, develop, test, and manage documents related to everyone on their staff, public safety officers and other.

These new offerings are just one small example of our commitment to our continued service to the public safety sector. As the dedicated men and women in the service of the public safety are required to possess more varied skills and talents, we at McCann are determined to help develop and assess those skills to ensure that the front lines of the public safety industry are ranked with men and women of the highest caliber. We look forward to another fifty years of growing with this community.

What are the benefits of computer adaptive testing (CAT) and why does CAT require such a large bank of items?

Unlike a linear test that administers the same set of questions to candidates either in the same sequential or randomized order, computer adaptive testing (CAT) delivers a unique set of test questions to each individual based on each individual's own ability level. Using a three parameter IRT model, McCann's CAT reaches into the entire item bank and, based on previously submitted correct and incorrect responses, pulls the next best item for the individual test taker. With advanced algorithms and statistical programming, CAT accurately measures each test taker's ability level across what is technically referred to as the "theta" scale, a fancy way of saying "ability scale".

Benefits of CAT include reducing item exposure since typically no two individuals see the same set of test questions, more accurately pinpointing individual ability levels, and more quickly reaching valid judgments with respect to whether a candidate has demonstrated minimal competence of the subject matter. 1

However, CAT is not right for all assessment programs and careful analysis and scrutiny should be undertaken when deciding whether to move from the more traditionally administered linear testing to CAT. First, CAT is only a consideration for computer-based or online-delivered assessments. It doesn't work, for good and practical reasons, for paper/pencil delivery of assessments. Second, CAT may impose a considerable expense in item development since, as a general rule, the total number of items in the bank should be at least six to ten times as large as the items delivered on the tests. Third, CAT usually involves extended bank development at the "tails", that is, CAT requires more "very easy" and "very difficult" items than what is normally required for linear tests. Fourth, CAT may or may not be the right solution for some diagnostic assessments where the intent and purpose of the assessment is to provide detailed feedback to candidates on strengths and weakness across many content domains. While our technology has been used to deliver millions of adaptive tests each year, we also deliver testlets, branching profiles, and linear tests for various clients situated across the world. My staff and I enjoy working with clients to create the best program for each one, including the best delivery mode for each one, depending on many criteria such as current item bank status, number of test takers annually, budget and time requirements.

1Bergstrom, B., & Gershon, R. (Winter 1994). Computerized adaptive testing for licensure and certification. CLEAR Exam Review, 25-27.

What's the difference between certification and licensure?

Many people use these terms interchangeably, but there are differences. Important and substantial differences.

The federal government has defined "certification" as the process by which a non-governmental organization grants recognition to an individual who has met predetermined qualifications specified by that organization.1 Similarly, the National Commission for Certifying Agencies has recently defined certification as "a process, often voluntary, by which individuals who have demonstrated the level of knowledge and skill required in the profession, occupation, role, or skill are identified to the public and other stakeholders."2

Licensure, on the other hand, is the state's grant of legal authority, pursuant to the state's police powers, to practice a profession within a designated scope of practice. Under the licensure system, states define, by statute, the tasks and function or scope of practice of a profession and provide that these tasks may be legally performed only by those who are licensed. As such, licensure prohibits anyone from practicing the profession who is not licensed, regardless of whether or not the individual has been certified by a private organization.

1 U.S. Department of Health, Education, and Welfare, Report on Licensure and Related Health Personnel Credentialing (Washington, D.C.: June, 1971 p. 7). 2 NCCA Standards for the Accreditation of Certification Programs, approved by the member organizations of the National Commission for Certifying Agencies in February, 2002 (effective January, 2003). Excerpt from: http://www.nremt.org/about/Legal_Opinion.asp

What are assessment centers and why are they important in personnel selection?

Assessment centers refer to a specific exam process, not a building. An assessment center is a standardized process used for objectively evaluating job candidates using realistic scenarios or simulations that they are likely to encounter on the job for which they are applying. They typical measure skills that cannot be measured via a multiple-choice or other written item type. By way of example, some of the assessment center exercises we employ at McCann include structured oral interviews, role-playing scenarios, in-basket exercises, leaderless discussions, and other similar type of situational simulations. A panel of experts observe candidates performing the exercises and make objective judgments about the candidates' likelihood of effectively handling similar situations on the job. As you can imagine, assessment centers can be expensive, expensive to create, expensive to administer, and expensive to judge/rate/score.

So why use assessment centers?
  • Assessment centers measure skills and abilities that paper/pencil tests simply can't. Some personnel, certification and/or licensure programs contains some very important skills and abilities that can only be fairly and accurately measured through an assessment center activity.
  • Research has found that assessment centers can more accurately predict successful performance as supervisors and managers than alternate methods.
  • Research shows that assessment centers are fair to minorities and women.
  • Assessment centers can be used to identify training needs.

Got a question? Ask the expert. Submit your question here.