Basics Reliability and Consistency in Psychometrics By Kendra Cherry Kendra Cherry Facebook Twitter Kendra Cherry, MS, is an author and educational consultant focused on helping students learn about psychology. Learn about our editorial process Updated on May 07, 2020 Fact checked Verywell Mind content is rigorously reviewed by a team of qualified and experienced fact checkers. Fact checkers review articles for factual accuracy, relevance, and timeliness. We rely on the most current and reputable sources, which are cited in the text and listed at the bottom of each article. Content is fact checked after it has been edited and before publication. Learn more. by Sean Blackburn Fact checked by Sean Blackburn LinkedIn Sean is a fact-checker and researcher with experience in sociology and field research. Learn about our editorial process Print Kaan Tanman / Getty Images When we call someone or something reliable, we mean that they are consistent and dependable. Reliability is also an important component of a good psychological test. After all, a test would not be very valuable if it was inconsistent and produced different results every time. How do psychologists define reliability? What influence does it have on psychological testing? Reliability refers to the consistency of a measure. A test is considered reliable if we get the same result repeatedly. For example, if a test is designed to measure a trait (such as introversion), then each time the test is administered to a subject, the results should be approximately the same. Unfortunately, it is impossible to calculate reliability exactly, but it can be estimated in a number of different ways. Test-Retest Reliability Test-retest reliability is a measure of the consistency of a psychological test or assessment. This kind of reliability is used to determine the consistency of a test across time. Test-retest reliability is best used for things that are stable over time, such as intelligence. Test-retest reliability is measured by administering a test twice at two different points in time. This type of reliability assumes that there will be no change in the quality or construct being measured. In most cases, reliability will be higher when little time has passed between tests. The test-retest method is just one of the ways that can be used to determine the reliability of a measurement. Other techniques that can be used include inter-rater reliability, internal consistency, and parallel-forms reliability. It is important to note that test-retest reliability only refers to the consistency of a test, not necessarily the validity of the results. Inter-Rater Reliability This type of reliability is assessed by having two or more independent judges score the test. The scores are then compared to determine the consistency of the raters estimates. One way to test inter-rater reliability is to have each rater assign each test item a score. For example, each rater might score items on a scale from 1 to 10. Next, you would calculate the correlation between the two ratings to determine the level of inter-rater reliability. Another means of testing inter-rater reliability is to have raters determine which category each observation falls into and then calculate the percentage of agreement between the raters. So, if the raters agree 8 out of 10 times, the test has an 80% inter-rater reliability rate. Parallel-Forms Reliability Parallel-forms reliability is gauged by comparing two different tests that were created using the same content. This is accomplished by creating a large pool of test items that measure the same quality and then randomly dividing the items into two separate tests. The two tests should then be administered to the same subjects at the same time. How Do Personality Tests Work? Internal Consistency Reliability This form of reliability is used to judge the consistency of results across items on the same test. Essentially, you are comparing test items that measure the same construct to determine the tests internal consistency. When you see a question that seems very similar to another test question, it may indicate that the two questions are being used to gauge reliability. Because the two questions are similar and designed to measure the same thing, the test taker should answer both questions the same, which would indicate that the test has internal consistency. Does the Rorschach Inkblot Test Really Work? Influencing Factors There are a number of different factors that can have an influence on the reliability of a measure. First and perhaps most obviously, it is important that the thing that is being measured be fairly stable and consistent. If the measured variable is something that changes regularly, the results of the test will not be consistent. Aspects of the testing situation can also have an effect on reliability. For example, if the test is administered in a room that is extremely hot, respondents might be distracted and unable to complete the test to the best of their ability. This can have an influence on the reliability of the measure. Other things like fatigue, stress, sickness, motivation, poor instructions and environmental distractions can also hurt reliability. Reliability vs. Validity It is important to note that just because a test has reliability it does not mean that it has validity. Validity refers to whether or not a test really measures what it claims to measure. Think of reliability as a measure of precision and validity as a measure of accuracy. In some cases, a test might be reliable, but not valid. For example, imagine that job applicants are taking a test to determine if they possess a particular personality trait. While the test might produce consistent results, it might not actually be measuring the trait that it purports to measure. Types and Problems With Personality Testing 5 Sources Verywell Mind uses only high-quality sources, including peer-reviewed studies, to support the facts within our articles. Read our editorial process to learn more about how we fact-check and keep our content accurate, reliable, and trustworthy. Institute of Medicine. Psychological Testing In The Service Of Disability Determination. Washington: National Academies Press; 2015. Leppink J, Pérez-fuster P. We need more replication research - A case for test-retest reliability. Perspect Med Educ. 2017;6(3):158-164. doi:10.1007/s40037-017-0347-z Albers MJ.. Introduction to quantitative data analysis in the behavioral and social sciences. Wiley. 2017. Hu Y, Nesselroade JR, Erbacher MK, et al. Test reliability at the individual level. Struct Equ Modeling. 2016;23(4):532‐543. doi:10.1080/10705511.2016.1148605 Polit DF. Getting serious about test-retest reliability: a critique of retest research and some recommendations. Qual Life Res. 2014;23(6):1713-20. doi:10.1007/s11136-014-0632-9 Editorial Process Our Review Board Share Feedback Was this page helpful? Thanks for your feedback! What is your feedback? Other Helpful Report an Error Submit Speak to a Therapist Online Advertiser Disclosure × The offers that appear in this table are from partnerships from which Verywell Mind receives compensation.