Welcome

Welcome to Lelp

 Return to Introduction

Welcome to Lertap help, "Lelp".

This document is designed to be used online, in conjunction with the Lertap 5 item, test, and survey analysis system.

It may also be used as a stand-alone online reference document, that is, without Lertap 5. To assist with stand-alone mode, we've compiled this document in a couple of different formats.

Please direct questions or comments to: support@lertap.com.

Last update: 29 June 2009.


What is Lertap?

Lertap, the Laboratory of Educational Research Test Analysis Package, is a computer program used to process and analyse results from tests and surveys. The fifth version of Lertap, released in 2001, is designed to work as an application running within Microsoft's Excel program.

This document provides some idea of what Lertap does, and how to go about getting it to do what it does. If you're new to Lertap, you might find other resources to have a more introductory flavour. For example, there's the knock-your-socks-off manual, and, of course, the website.


Requirements

Lertap 5 is an Excel application. It requires a copy of Microsoft's Excel program to run in. To date Lertap has been tested with Windows 98, 2000, XP, and NT4. It has also been tested on Macintosh computers, using MacOS 8, 9, and 10.

In all cases, Lertap has been used on computers running English language versions of Excel. Non-English versions of Excel have at times caused problems; for example, the Chinese language version of Excel has posed problems for would-be Lertap users. On the other hand, Thai versions of Excel have been able to run Lertap.

Is it necessary for Lertap users to be expert users of Excel? No. Some knowledge of Excel can be helpful at times, but it's not required.


How to get it

ASC, Assessment Systems Corporation, is the sole distributor of Lertap 5. ASC's website is: www.assess.com. The Lertap area within the ASC website may be reached by entering LERTAP in the Search box found towards the upper left of ASC's home page.

Getting yourself fixed up with a copy of Lertap is not difficult. If you use an Apple computer, a Macintosh of some sort, then you can get a test copy of Lertap by sending a request directly to Lertap headquarters in Western Australia: support@lertap.com.

Windows users follow a different procedure: they pay a visit to the ASC Lertap area mentioned above, and download a "trial copy" to their computer.

As of June, 2009, the ASC Lertap area had its download options located close to the bottom of the main Lertap page. Scroll down on the page, and eventually you'll come to a section which looks like this:

There are two versions of Lertap for Windows, one for Excel 2002 & 2003, and another for Excel 2007.

Click on the appropriate "Download the 30-day trial copy...." link, and away you will be. Note that if you're not sure which option is the "appropriate" one, Excel 2002 & 2003 or Excel 2007, then click the link shown with Note 1 (and, please note that this is just a picture of the ASC page; if you try and click above not much at all will happen -- you have to go to the actual Lertap area within the ASC website, www.assess.com, and then do your clicking).

The trial copy may be used for free for 30 days. A license must be purchased if Lertap is to be used after the 30-day trial period. Click here to read about licensing matters.


How to install it

The trial copy of Lertap downloaded from the ASC site will arrive at a user's computer as a single compressed file in the "Zip" format.

The name of the file will be similar to: Lertap564_WindowsTrial.zip.

Within this zip file there will be found a corresponding "exe" file, similar in name to: Lertap564_WindowsTrial.exe. The exe file may be extracted from the zip in a variety of ways -- usually a simple double-click on the zip file will prompt Windows to call up some sort of extraction program. (Often a right-click on the zip file will uncover suitable extraction options; if these ideas fail, search the internet for "extracting zip files".)

Once the exe file has been extracted, double-clicking on it will get the installation process started.


How to license it

A license to use Lertap after the 30-day trial period must be purchased from ASC. Please click here for further comments.


Known problems

A list of the most frequent problems related to installing and running Lertap may be found at this URL:

http://www.lertap.curtin.edu.au/KnownProblems.htm

Of the known problems, one merits special mention as it crops up so often: the matter of "Excel macro security". This problem is highlighted below -- it's very easy to solve. A far less common problem, but a worrisome one nonetheless, has to do with a 255-character limit in Excel cells; this problem is also mentioned below.

Excel macro security

Fairly often users report that they have successfully downloaded the compressed Lertap file from ASC, and then gone on to extract and install the various files which comprise the complete Lertap system.

However, when they try to start Lertap, all they see is this:

The problem with the screen snapshot shown above is that the Lertap toolbar is not showing. The screen should look like this:

The Lertap toolbar is now showing -- it's the toolbar with the yellow smiley face towards the left ToolbarIconSmileyFace.

How to make sure the Lertap toolbar is displayed when Lertap starts? Make sure Excel's macro security level is set to Medium or Low, not High. Click here to read how to do this.

Problems with very long *col lines

If you have a browse of Excel's Help file, you might find that it says any single worksheet cell may have some 32,000 text characters. It turns out that this is and isn't accurate -- some very practical problems can arise whenever a cell contains more than 255 characters. For Lertap users, this 255-character limit is something to keep in mind when using long *col lines in the CCs worksheet. Whenever Lertap finds a *col line which exceeds 255 characters, it displays a special message, explaining that Excel will not let it process the line, and requesting that the problem be fixed. How? We've got a special document with three suggested solutions. It's a Word file, to be found at this URL:

http://www.lertap.curtin.edu.au/Documentation/ExcelColumnLimitProblem1.doc


Contact us

Lertap is a project of Curtin University of Technology. Larry Nelson is the project director.

The project's website is at:

www.lertap.curtin.edu.au

Address:

(Dr) Larry Nelson

Lertap Project

Faculty of Education, Language Studies, and Social Work

Curtin University of Technology

GPO Box U-1987

Perth, Western Australia 6845

Phone | +61 8 9266 2183
Fax | +61 8 9266 2547
email | L.Nelson@curtin.edu.au


About this document

This document, "Lelp", was made using Help&Manual 4.1, a hypertext authoring system from EC Software.

H&M3 lets us compile Lelp in one of several formats, and we've done so. To date there are PDF and chm compilations. "chm" help files are now found in most Windows applications; they're easy to use. PDF files are now a world standard, usable on all sorts of computers; this is nice, but PDF files cannot display what are called "popup" topics. This document contains quite a number of popups -- they flash up quickly in the chm version, but don't show at all under PDF. If you're looking at the PDF version, you'll sometimes see spots which say something like "click here", or "note", but there's nothing clickable -- alas, you've come upon a popup that won't pop.

Lelp is also available as a website. The address is:

http://www.lertap.curtin.edu.au/HTMLHelp/HTML/index.html

Since all forms of this document are produced from the same source, they're identical. The chm, PDF, and website versions have exactly the same content. However, as noted, the PDF version cannot display popup topics.

The screen shots seen in this document, Lelp, were taken when Lertap was running under Microsoft's Windows XP operating system, using Excel 2002.

Screen shots in manuals have a habit of dating quickly; the shots you see in this document will not have a 100% correspondence with the screens seen in the most recent version of the Lertap 5 system. This is particularly true for users with Macintosh computers. There are differences between Windows and Macintosh system icons -- these are most noticeable when it comes to toolbars and menu options (the Lertap toolbar looks quite different on a Mac, for example).

Related tidbit:

Copies of the various versions of Lelp are freely available at the website:

http://www.lertap.curtin.edu.au/Documentation.htm


Differs from manual.

Lelp, that is, this document, serves a variety of purposes.

Above all, Lelp effectively updates the manual. For a summary of the major changes made since the manual was printed, see Lelp's Revisions topic.

A number of supporting documents have been made available to users since the manual came to light, and Lelp has links to them, often as URLs found under the "Related tidbits" section at the end of some topics.

All versions of Lelp, chm, PDF, and website, may be viewed online. The manual is also available in an electronic form, but, unlike Lelp, it's not cross-referenced, it has not been formatted as hypertext -- it's not as easy to jump from topic to topic in the manual as it is in Lelp.

Both the manual and Lelp have numerous examples. The ones in Lelp tend to be somewhat more advanced.

Lelp's explanation of CCs lines and syntax is somewhat more extensive than that found in the manual.

Users of the Windows version of Lertap 5 will find that some of Lertap's dialog boxes, and most of its menus, have automatic, context-sensitive links to Lelp. By and large, these links are denoted as "Lelp". Macintosh users do not yet have a similar resource -- they may refer to the website, or to the PDF version.

Finally, this document has an invaluable educational supplement: in numerous spots it introduces elements of Australian slang (strewth!); in other spots it has links to selected Western Australian cultural highlights, such as Emu Export, Mt Barker, and the Southwest Capes.


Updates

This document is updated much more often than its stable mate, the Lertp5.xls file.

Check the website for updates. They're freely available at:

http://www.lertap.curtin.edu.au/Documentation.htm

Windows users will probably want to download the chm file. (Of the various formats this document comes in, the chm format is likely to be the handiest for Windows users -- it's a conventional Windows Help file.) The full name of the chm version is

LRTP5HHelp.chm

For Windows users, a copy of this file is automatically installed when Lertap is downloaded from ASC. To find out where it installed to, use Windows' Start menu to Search for it. Once you've found where's it's been parked by the ASC setup program, you can replace it with the latest version, as obtained from the website.

This discussion may prompt you to ask how to tell if you've got the latest version. The date of the latest version may be found by looking at the Welcome page on the website. Go to this URL:

http://www.lertap.curtin.edu.au/HTMLHelp/HTML/index.html

The Welcome page is the first page. Compare the date found at the bottom of the website's Welcome page with that found on the version you've been using, and Bob's your uncle!


About Lertap version

To find out which version of Lertap you're running, and its date, click on the yellow smiley face on Lertap's toolbar.

We update from time to time. Major updates, referred to as revisions, or as "salient changes", are summarised in Lelp, the document you're reading at this very moment. Click here to branch out to Lelp's Revisions topic.

Most of the updates are quite minor, and are not classed as revisions. You can see a summary of them by clicking here if you're online:

http://www.lertap.curtin.edu.au/Documentation/UpdatesSummary.htm

Upgrades: It's not hard to get an updated version of Lertap once you've licensed it. Before you think about upgrading, re-read the comments immediately above -- you may not really need an upgraded version -- your present version may be recent enough to meet your needs.

But, if your little heart thinks an upgraded Lertap is something you just have to have, then, to avoid heartbreak, read on ...

If you use a computer running a version of Windows, or NT, and if you have either licensed Lertap, or are still running with an unexpired 30-day trial copy, then you have two options for upgrading Lertap.

Option 1. Visit the Lertap area at Assessment Systems Corporation's (ASC) website (www.assess.com), and download the Trial Copy for Windows. Yes, this is billed as the 30-day trial copy, but that's okay: download it, and install it. The installation process will detect the presence of a Lertap license and will apply the license to the just-installed version.

Option 2. Visit the Lertap website at Curtin University. Go to the Software page. Scroll to the bottom of the page. Carefully read what's been written there (carefully!). Download the LRTP5.zip file. Unzip it. Rename LRTP5.xls as Lertap5.xls. If you want, you can search your computer to find out where it has your original copy of Lertap5.xls, and replace the original with what you've just downloaded. (Hint: you may find it under C:\ASC\Lertap56, or perhaps under D:\ASC\Lertap56.)

You gotta keep this in mind: the two options above only work if you're using a Windows computer, and then only if you have previously installed a version of Lertap on it. If you see the "Whoopsie" message, and an announcement that you're missing "an important file, KeyLib32", then you're up the creek without an implement capable of moving you forward (translation: the Lertap version you've just downloaded is not going to work).

What's the difference, if any, in downloading from Assessment Systems and downloading from the Curtin University website? What makes the two options above different? Answer (1): the Curtin University website, Option 2, is sure to have the most recent version of Lertap -- it's only when we've come out with one of those earth-shaking revisions mentioned above that the version at Assessment Systems is upgraded.

Answer (2): the ASC option, Option 1, does a more thorough job of upgrading your (Windows) computer: when you first install Lertap, using the download from ASC, the ASC setup program does the right thing with Windows: it registers Lertap as an "application", it sets up icons to quick-launch Lertap, and it makes Lertap available through the Start button's Programs list. Every time you subsequently download Lertap from ASC, the same thing happens, making for a typically complete Windows program installation.

Not so with Option 2: all you end up with, should you take this option, is simply a file, Lertap5.xls. To run this file, you have to open it, and, of course, you have to know where it is. It's real important to keep this in mind; if you've gone with Option 2, then using the Windows Start button to locate Lertap in the Programs listing will activate the old version of Lertap, the one you last downloaded from ASC. In order to activate the new version, the Lertap5.xls file resulting from Option 2's steps, you have to find the file and open it, something usually done by double-clicking on the file name.

If you've taken Option 2, and your Windows skills are up to it, you could replace the copy of Lertap5.xls installed from the ASC download with the copy of Lertap5.xls from Option 2. How? You'd use the Windows Start button to Search for Lertap5.xls. It should return two hits -- the old (ASC) version, and the new (Option 2) one. You'd want to replace the old version with the new one; if you do this, then you can run the new version via the Start button's Programs list.

If your new Lertap version hums along okay, then you might also want to upgrade your copy of Lelp. To read about how, give a little click here, if you will.

What about Mac users who want to update their Lertap version? Best write to us at support@lertap.com.


Getting started

It's a fairly simple matter to get started with Lertap. Let us get you launched on the so-called "Cook's tour", and you can see for yourself.

When you start Lertap, your computer screen should display a window which looks something like this one:

One of the most important parts of the screen above is the Lertap toolbar. If you can see the toolbar, you're on track for a successful tour. If you can't see the toolbar, check the known problems topic for help. You must be able to access the toolbar -- it's the essence of Lertap.

Another core component of the screen showing above is the line of worksheet tabs showing towards the bottom of the screen. There are four tabs: Comments, Data, CCs, and Syntax. (Revision note, June 2004: there's now a fifth tab: System.)

Each tab corresponds to an Excel worksheet. The Data worksheet has the responses of 60 people to two quizzes. The CCs worksheet has a series of lines with Lertap's control syntax. Each of the syntax lines gives Excel information on how to process the data found in the Data worksheet. The Syntax worksheet is a quick reminder for experienced Lertappers on the correct format for CCs lines.

Go ahead and look at each of the worksheets if you'd like. Then come back here.

Okay? The cook's tour starts with the New menu. Go to the Lertap toolbar, and click on New.

Click on the option which says "Make a new Lertap 5 workbook which is a copy of the present one." This causes Lertap to do a couple of somersaults and back flips -- it makes a copy of the Data and CCs worksheets, and places them in a new Excel workbook. When it finishes this task, Lertap says something like this:

Don't worry about saving the new workbook for the moment. Click the OK button.

Once you've done this, click on the Run menu. Go to the Lertap toolbar, and click on Run.

Click on the line which says "Interpret CCs lines". This gets Lertap to read the lines in the CCs worksheet, checking to make sure they have the right syntax. If they do, Lertap creates some new worksheets, and displays one of them: "Freqs".

The Freqs worksheet is a simple one. It displays what are called "response frequencies" for each of the columns in the Data worksheet.

The little boxes above indicate that 26 people answered "A" on Q1.

The main purpose of the Freqs worksheet is to see if there may have been any errors in the processing of the data. Users generally scroll down the Freqs worksheet rather quickly, looking for unexpected results. For example, a response of "D" on Q1 would be strange as Q1 allowed for just three responses,A, B, and C.

To continue the tour, return to the Run menu now, and click on "Elmillon item analysis". This causes Lertap to do quite a number of things -- for example, it creates test scores, putting them in a new worksheet called, appropriately, Scores. It also creates a variety of statistical reports, giving these names such as Stats1f, Stats1b, Stats1ul, Stats2f, and Stats2b. The partial screen snapshot below shows what the worksheet tabs will look like after this step:

The various "Stats" worksheets are usually why people run Lertap -- they give item and test statistics. Stats1f provides a complete, or "full", item and test analysis report for the first quiz, or test, giving quite detailed information for each test item, and presenting several mini-reports with overall test statistics.

The Stats1b report is a briefer summary of the item statistics found in Stats1f; it's easier to read. If the test being analyzed is a cognitive test, Lertap generally produces a third report, Stats1ul; the "ul" means upper-lower, referring to a method of item analysis preferred by some.

Lertap produces more reports than many people want. For example, the "b" Stats sheets, such as Stats1b and Stats2b, are redundant in that their information is also to be found in the respective "f" sheets, such as Stats1f and Stats2f. Some users make little use of the "b" sheets, preferring to drink their cup of tea with the "f" sheets. On the other hand, some Lertap users find there to be too much information in the "f" sheets; their needs are met by the "b" sheets, where they find that just half a cup of tea, a "brief" cup, is all that's needed to peruse the output. (It's possible to get Lertap to reduce the number of reports it gives. This is done by setting options in rows 9 and 10 of the System worksheet.)

Why are the "f" sheets made so that their initial focus is well down the worksheet? Stats1b and Stats1ul, for example, display their top rows at the top of the screen, but Stats1f and Stats2f do not display their upper-most rows at first. Why? Because experience has shown that the majority of users scroll to the reliability section of the "f" sheets before they look at anything else. A similar effect is seen in the Scores worksheet, where Lertap places the initial focus well down the worksheet, where the summary statistics begin.

For a thorough discussion of the statistics found in the Stats sheets, please refer to the manual.


About the Cook's tour

We might do well to highlight some of the things that happened when you took the "Cook's tour" covered in the previous topic. And, as you'll read below, we've made some changes to Lertap which result in the tour producing more reports.

The Cook's tour is based on the Lertap Quiz data set. This data set is fully described in Appendix A of the manual. Briefly, the Quiz consists of a "test" given to 60 people who had the good fortune to participate in one of the very first Lertap workshops, held in Dunedin, New Zealand, just a couple of years ago (it was 1973! -- for us time flies when we're fishing, camping, or Lertapping).

At the end of the workshop, each participant was asked to answer 25 multiple-choice questions and 10 Likert-style affective questions.

The multiple-choice questions were meant to indicate how well the participants had mastered the content of the workshop -- the 25 questions were a test of their knowledge of the functioning of Lertap.

The 10 Likert questions, scored on a 5-point strongly-agree to strongly-disagree scale, asked the participants how they felt about Lertap -- did they judge it to be the dynamite test and survey analysis system we know it to be?

Each participant was also asked to respond to two open-ended questions. One of these asked them to state how long they had been using computers, while the other requested information on how long they had been using tests in their research or teaching.

The Lertap Quiz data set is included in the Lertap5.xls file. When you start Lertap, you'll see tabs at the bottom of Excel's screen, one for each of the worksheets contained in the Lertap5 workbook (refer to the previous topic for a picture). If you look at the Data worksheet, you'll see the actual question responses given by the 60 participants in the 1973 workshop.

When you took the tour, your first action was to get Lertap to make a new data set for you to fool around with. This you did by going to the New menu, clicking on the option which directed Lertap to make a new workbook by creating a "copy of the present one".

The "present one", in this case, referred to the data set which comes with the Lertap5.xls file, that is, to the Lertap Quiz. At the end of this step you had a new workbook which contained copies of the Data and CCs worksheets corresponding to the Lertap Quiz. All subsequent actions in the cook's tour involved use of these copied worksheets.

Changes to the CCs lines (2004)

In the September 2004 revision of Lertap, we provided an option to process affective questions in a different manner, as requested by users in Australia and the United States.

The revision involves vesting more flexibility in the "MDO" option. Read about the new MDO's capabilities by simply clicking here whenever you'd like.

To demonstrate how the revised MDO works, we have modified the CCs lines related to the Lertap Quiz. Now the lines set out three subtests, not two as was formerly the case.

If your copy of the Lertap5.xls file is dated 12 September 2004, or later, then your cook's tour will be a bit different: you'll find two new reports have been added to former output, Stats3f and Stats3b by name. The report goodies provided by the new MDO are almost all showcased in the Stats3b report. Click here to see a sample.


How it works

You've taken the cook's tour? Goodonya (!).

We've got some examples coming up, but first how about a quick overview of how Lertap works?

Lertap is used to analyse the responses people have given to a test or survey. Lertap is designed to work with fixed-choice items (items = questions). For a test, fixed-choice usually means true/false or multiple choice. For a survey, fixed-choice means that the answers people can give to a question have been listed, and enumerated. For example, a survey might ask people if they think beer is a good thirst quencher on a hot summer's day; people can answer 1 (strongly disagree, a foolish response); 2 (disagree); 3 (undecided); 4 (agree); or 5 (strongly agree).

The answers people give are placed in the Data worksheet. Each row in the Data sheet corresponds to the answers of one person.

Having responses recorded in the Data worksheet is good and necessary before results can be obtained, but it's not all that's required. Some instructions are also needed. Instructions? For whom? For Lertap.

Lertap is not smart enough to be able to look at the Data sheet, and figure out what's what without your help. You've got to tell Lertap which columns in the Data worksheet have the item responses you want it to look at. If the items are from a test, you have to tell Lertap what the right answer to each item is. If the items are from a survey, on the other hand, you have to tell Lertap to forget about having to have a correct answer for each question -- there aren't any.

You tell Lertap these things by putting your instructions in another worksheet called the CCs sheet, using a special control language.

Once you've got the responses in the Data sheet, and your Lertap instructions in the CCs sheet, away you go ... you just follow the same steps seen in the cook's tour. Could life be simpler? Page on ... have a look at some examples (there are more in the manual).

Related tidbit:

For more how-it-works insights, see: Lertap's Output.


Examples

We'll present several samples for you to cuddle. The first will be a simple 10-item cognitive test. The second will be a simple 12-item survey. Then we'll dramatically step up the action by looking at the sample data set which comes with Lertap -- a data set with 25 cognitive items, and 10 survey questions. Got a big, fresh cuppa ready? Go ....


Cognitive example

Lookit, lookit, lookit ... here's a Data worksheet ...

What have we got here? Fifteen students answered 10 multiple-choice questions. On the first item, Q1, Anderson selected option D. On the fifth item, Invererity selected option B.

Anderson's data are found in row 3 of the worksheet. The answers to Q1 are found in column 2 of the worksheet. The answers to the last question are found in column 11. (You can't see row 18, but it's empty.)

And here's the corresponding CCs worksheet:

Yes, the CCs worksheet has just two rows of information. The first one tells Lertap that item responses are to be found in columns 2 through 11 of the Data worksheet. The second line has ten letters; each letter is the right answer, the "key", for an item. For example, the right answer to the first question is B. The right answer to the 6th item is C. The correct answer for the 10th item is B.

And that's it. Yes. The data set is ready for Lertap. At this point, one would go to Lertap's Run menu and click on "Interpret CCs lines". Then, after being Freq-ed out, we'd go back to the Run menu and click on "Elmillon item analysis". Beauty.

Related tidbit:

The example above is based on a cognitve data set which may be seen at this URL:

http://www.lertap.curtin.edu.au/Documentation/Samples/TenCogs/TenItemCognitive.doc


Affective example

Here's another example:

Here we've got another 15 students, and this time we have responses to 12 survey items. Some sort of number is found in the first column, with the responses to the first question, Q1, found in the 2nd column.

Once again there are only two CCs rows with information. The first row tells Lertap that item responses are found in columns 2 through 13 of the Data worksheet. The second row tells Lertap that these items are "affective", or survey, items. Having seen the word "affective" on a *sub CCs line, Lertap knows that the questions do not have a correct answer -- they're to be scored using the default survey scoring scheme where a response of 1 equals one point, a response of 2 = 2 points, and so on.

This example is complete. Nothing is missing. At this point, one would go to Lertap's Run menu and click on "Interpret CCs lines". Then, after being Freq-ed out, back to the Run menu and a click on "Elmillon item analysis". Beauty abounds, doesn't it?

Related tidbit:

The example above is based on a sample survey data set which may be seen at this URL:

http://www.lertap.curtin.edu.au/Documentation/Samples/CEQ/CEQ1.DOC


Lertap data set

We've given you two really straightforward examples. In our experience, the two simple examples you've seen would be very much like what maybe a quarter of Lertap users regularly live and breathe. But Lertap was designed to handle more complex situations. Those little CCs lines can pack more punch than what's been on display in the previous two samples.

As an example, look at this CCs worksheet:

There are 11 lines in use in this CCs example. Four (4) of the lines are comments; these are the lines which do not begin with an asterisk. The use of comments is entirely optional, but they can be real helpful.

There are two *col lines above. Each of these defines a group of items which will be processed together, as a unit. Such units are generally referred to in Lertap as "subtests".

A subtest may be comprised of cognitive items, or it may be comprised of affective items.

You've already seen examples of the CCs lines used with these two types of subtests. We've pointed out that cognitive subtests will always have a *col line and a *key line, while affective subtests will always have a *col line and a *sub line.

In the example above, a *sub line has been used with the cognitive subtest for several reasons. In this example, some of the items had as many as six possible responses -- that is, some of the multiple-choice items used in this subtest had six choices, or responses.

The Res=(A,B,C,D,E,F) declaration tells Lertap this. Without an Res= declaration, Lertap assumes Res=(A,B,C,D), the default set of item response codes for cognitive items. (The default set for affective items is Res=(1,2,3,4,5).)

The *key line gives the right answer for each of the 25 items in the first subtest. The *alt line tells Lertap that the items used a different number of the six possible responses. For example, the first item used just 3 responses: they would be A, B, and C, the first three characters found in the Res= declaration. The second item used 5 responses: A, B, C, D, and E. Only two of the 25 items made use of all 6 possible responses.

The Name= declaration provides a brief description of each subtest, while the Title= declaration gives a short title. When Name is used, some of Lertap's reports display Name on their top line. Title, when used, appears in some reports as a label for subtest scores.

Subtest scores? What are they? Well, as an example, on a multiple-choice test people usually get one point for each right answer. On a 25-item test, or "subtest", it would be possible to get a score of 25, assuming one point for each correct answer (Lertap permits the right answer to have any number of points, and it even allows the wrong answers to have points too -- sometimes wrong answers are penalised by assigning them negative points).

Let's look now at the affective subtest defined above. The *col line points to 10 (ten) columns, c28 through c37. A *sub line is required for affective subtests, and it must have the "Aff", or "Affective", control word on it. This is seen above. In this case, the *sub line has also been used to assign a Name and a Title.

What's that *pol line doing? To answer this question, we have to return to the matter of scoring. Affective items do not have a correct answer. It's customary to give a certain number of points for each of an affective item's answers.

What were the possible response choices for this set of 10 affective items? Very good question. There is no Res= declaration on the *sub line, and, in this (common) case, Lertap assumes Res=(1,2,3,4,5). Each affective item had five possible responses. Unless you say otherwise, Lertap will give one point if someone selects 1 as their response; two points when someone selects 2; and so on. This is called "forward scoring". On a 10-item affective subtest with five response choices per item, the maximum possible score would be 50; the minimum possible would be 10.

The *pol line allows each affective item to be reverse-scored, if wanted. On a reverse-scored item, the first possible response will get 5 points, not 1. The last (fifth) response will get just 1 point, not 5. Reverse scoring for affective items is pretty common.

The *pol line above indicates that the first item is to be forward-scored, while the next four items are to be reverse-scored. This subtest has 10 items, so there are 10 + or - (minus) symbols shown on the *pol line. (In fact, for this subtest, four items are forward-scored, while six are reverse-scored.)

Lertap's forte is in the flexibility it provides for item scoring. Any response to any item can have any "weight", that is, any number of points. In the three examples we've presented thus far, including the one above, items are being scored in a conventional manner. Departures from normal are supported by the use of other CCs control lines, such as *wts and *mws.

Are we going to get away without talking about the Wt= declaration seen on the two *sub lines? No siree Bob; here goes: whenever multiple subtests are scored, Lertap will add up all the subtest scores to get a total, or "composite", score for each person. Each subtest ordinarily comes into the composite with a weight of one (1); to keep a subtest out of the total score, Wt=0 is used. In the example above, both subtests have been given a weight of zero, and Lertap will not make a total score.

One final point which people often ask about ... there are spaces in the *key line above, in the *alt line, and also in the *pol line. There's a space after every five characters in each of these lines. Why? Simply to make the line a bit more legible. The spaces are not required.

Is the example above a common one? Yes and no. Yes in the sense that Lertap users frequently have more than one subtest to process, no in the sense of mixing subtest types -- this example has a cognitive subtest, and an affective subtest: a mix of subtest types -- that's quite uncommon. If you browse on into the following topics, you'll see a couple of other examples.

Related tidbit:

For a really bonza example of a job which worked Lertap's CCs lines close to the limit, have a look at "Using Lertap in a Parallel-Forms Reliability Study", a 16-page Word document available via the Internet: click here if you're connected.


Multiple cognitives

Multiple cognitive subtests

Consider these CCs lines:

*col (c1-c10)

*sub name=(Addition), title=(Add)

*key DCCAB BCDDA

*col (c11-c20)

*sub name=(Subtraction), title=(Sub)

*key BBBCA DAACB

*col (c21-c30)

*sub name=(Multiplication), title=(Mult)

*key CDCAB AAACC

*col (c31-c40)

*sub name=(Division), title=(Div)

*key AADCC CBAAA

Someone's given a maths test with four subtests. Each subtest had ten items.

Lertap will create four subtest scores, and a total score. The total score will simply be the sum of the four subtest scores. The maximum possible score on each subtest is 10, hence the maximum possible total score is 40. (It is possible to change the number of points given for right answers by using *mws and *wts lines in the CCs worksheet.)

The *sub lines do not have Res= declarations, so Lertap will assume Res=(A,B,C,D) for each subtest.


Multiple affectives

Multiple affective subtests

Here's a common example of CCs lines for a survey with three subscales:

*col (c5-c20)

*sub aff, title=(Anxiety)

*pol ++--+ -++++ ++-++

*col (c21-c35)

*sub aff, title=(Friends)

*col (c36-c50)

*sub aff, title=(Homesick)

*pol ---++ +++-- -+-++

Three 15-item affective subtests are defined by these lines. Two of the three subtests, the first and the third, have a mixture of forward- and reverse-scored items. There is no *pol line for the second subtest, which means that all items for this subtest are forward-scored.

Lertap will make three subtest scores, and also a total score.

What about possible score ranges for this example? Each subtest has 15 items. There being no Res= declaration on the *sub lines, Lertap assumes Res=(1,2,3,4,5), that is, five possible responses per item.

Lertap will score each item on a one- to five-point basis. Why? Because there are five possible responses. The minimum score a person can get on an item is one; the maximum is five. There are 15 items in each subtest. Therefore, the score range for each subtest is 15 to 75, and, there being three subtests, the range for the total score will be 45 to 225.

What happens when a person doesn't answer an item? What sort of score do they get?

For cognitive tests, a non-answer gets a score of zilch (zero). However, for affective items, a non-answer will get a score equal to the mean of the item's response weights (note). It is possible to defeat this scoring system by using the MDO control word on a *sub line.

Note that it is possible to achieve almost any sort of scoring for affective items, or, for that matter, cognitive items. This is done by using *mws lines in the CCs worksheet.


CCs details

You've seen that Lertap analyses start with the creation of two Excel worksheets: Data and CCs.

The CCs worksheet contains the all-important lines of Lertap syntax which effectively control how Lertap and Excel analyse the data found in the Data worksheet.

CCs really stands for Control Cards. It used to be the case that data analysis was based on the use of punch cards. Years back, a typical data analysis job involved the use of a keypunch machine -- data were punched on cards, as were the instructions which told the computer how to analyse the data. The first versions of Lertap were based on the use of punch cards, and the term "control cards" has been carried into most subsequent versions.

You'll see that we sometimes refer to the rows in the CCs worksheet as "lines", and sometimes as "cards". We use these terms interchangeably; they mean the same thing.

There are a total of eight "cards" which may be used in a CCs worksheet. The number of cards used in any given job depends on two main factors: the type of subtests being processed, and the complexity of the item scoring desired.

We'll spell out the general nature of all of the cards below. The topics immediately following provide more exact details on the syntax of each card.

*col | The basic Lertap control card, used and required by all subtests, cognitive and affective. Each time Lertap sees a *col card in the CCs worksheet, it thinks "Ah-ha, here comes a new subtest", and it expects the user to then indicate the columns in the Data worksheet which are to be processed. (Click here to read about a problem which can arise with very long *col lines.)
*sub | This card is optional for cognitive subtests, but required for affective subtests. *sub cards are used to convey particular subtest characteristics to Lertap, such as the name and title of the subtest, and the number and nature of the response codes used by the items belonging to the subtest.
*key | Gives the right answer for the items of a cognitive subtest. This card is always required for cognitive subtests, but it's not used at all with affective subtests.
*pol | "pol" stands for polarity, that is, for plus (+) or minus (-). Not used by cognitive subtests, and optional for affective subtests. When used, it defines the type of scoring to be applied to affective items: plus (+) for forward, and minus (-) for reverse.
*alt | An optional card for both cognitive and affective subtests. When used, it indicates the last response code used by each item. If this card is not used, it is assumed each item uses the same number of response codes. (The format of this card changed early in 2005: please refer to Example C7 under the Cognitive CCs topic for details.)
*wts | An optional card for cognitive subtests; not used by affective subtests. This card makes it possible to quickly tell Lertap that the items of a cognitive subtest have different "weights", that is, the right answers to the items have differing point values (the first question might be worth one point, for example, while other questions might be worth more points). This card is also known as the *wgs card.
*mws | An optional card for both cognitive and affective subtests. "mws" stands for multiple-weights specification. This is the most powerful control card of all -- it allows any weight to be applied to any item response.
*exc | An optional card for both cognitive and affective subtests. "exc" stands for exclude. This card is used to quickly remove items from a subtest. (*exc is not mentioned in the manual.)
*tst | This is a very special control card. It can only be used once in any CCs worksheet, and, when used, it has to be the very first card. It's used to get Lertap to make a copy of the data set, with only certain data records to be copied to the new data set's Data worksheet. *tst is used to set up a new Lertap workbook containing a subset of the original Data records (for example, just the males, or only those in a specified School District).

This has been a quick introduction to Lertap's control "cards". We'll go on now to provide more specifics, and we'll do this by subtest type, cognitive first, then affective.


Cognitive CCs

Cognitive control cards

Before getting into the syntax for CCs cards used to analyse cognitive items, let's come to terms with some terms.

Each cognitive item may use up to twenty-six (26) response codes. Response codes are also known as alternatives, or as options. A true/false item may use {T and F} as response codes, or {t, f}, or {1, 2}. A cognitive item with four possible responses may use codes of {A, B, C, D}; or {a, b, c, d}; or {1, 2, 3 , 4}.

Associated with each response code is a weight, the number of points a person gets for choosing the corresponding option. For example, if the right answer to an item is A, then people who select A will get a certain number of points; people who select one of the item's other responses will (usually) get no points.

Okay? Now then ....

Let's say we've given a 5-item cognitive test, with answers appearing in columns 2, 3, 4, 5, and 6 of the Data worksheet. We'd like Lertap to spin its magic, to analyse our data. In order to do this, we'll need to enter some lines in the CCs worksheet. Put on a fresh pot of coffee, pour yourself a cup, and have a look at the examples below.

Example C1:

This set of two CCs cards might be all that's required to get Lertap to analyse the data:

*col (c2-c6)

*key ACCDB

Anyone who selects A on item 1, C on items 2 and 3, D on item 4, and B on item 5 will get a score of 5 -- one point for each answer. Why? There are five items; the right answers, the "keyed-correct" answers, are shown on the *key card above. Unless you say otherwise, Lertap awards one point for each right answer.

Example C2:

We'll add a *sub card in order to have Lertap label some of its reports:

*col (c2-c6)

*sub Name=(Followup TV9 news quiz), Title=(NewsQuiz)

*key ACCDB

The *sub card is usually optional for cognitive subtests. Here we're using one just to give a Name and a Title to the subtest. The Name will appear as a heading at the top of Lertap's item analysis reports, such as Stats1f and Stats1b. The Title will appear at the top of one of the Scores columns, making it a bit easier to interpret the Scores report. The Name can have any length, but Title should be no longer than 8 characters. If Name and/or Title are not given on a *sub card, Lertap will create default labels: Name=(Test 1), and Title=(Test1).

Example C3:

Next we'll use a *sub card in order to turn on certain scoring options:

*col (c2-c6)

*sub Title=(NewsQuiz), PER, SCALE

*key ACCDB

Now the *sub card has three control words, Title, PER, and SCALE. PER gets Lertap to create a percentage score for each test taker, being the student's score expressed as a percentage of the maximum possible score. For example, if the maximum score is 5, and a student got three items correct, PER=60%. The SCALE control word adds the student's z-score to the Scores report; on a test with a mean of 3, standard deviation of 1, a student test score of 4 would correspond to a z-score of +1.00.

Example C4:

To switch Lertap into its mastery scoring and report mode, include the word MASTERY on the *sub card, as shown here:

*col (c2-c6)

*sub Title=(NewsQuiz), MASTERY

*key ACCDB

Using the MASTERY control word on *sub causes two things to happen. Each student will have her/his percentage score automatically included in the Scores report, just as happens when the PER control word is used. More importantly, the MASTERY control word gets Lertap to substantially alter one of its main statistical reports. The Stats1ul report will include a summary group statistics table, a variance components analysis, and two classification accuracy indices (please refer to Chapter 7 of the manual for details, and also take in a 2007 journal article dealing with the use of cut scores).

Lertap assumes the mastery cutoff percentage to be 70%. This can be reset quickly, as shown below:

*col (c2-c6)

*sub MASTERY=80, Title=(NewsQuiz)

*key ACCDB

The cutoff percentage has now been set to 80%. More generally, it is possible to have the default level of 70% set to any value by making a change in Lertap's System worksheet. (Click here if interested.)

Example C5:

This example reflects a common situation:

*col (c2-c6)

*sub Res=(1,2,3,4), Title=(NewsQuiz)

*key 13342

The RES control word is telling Lertap that the item response codes are digits, not letters. Unless you tell it otherwise, Lertap assumes that cognitive items have four options, with response codes of {A, B, C, D}. If this is not the case, you must use an Res=() declaration on a *sub card, as exemplified above. Note that the *key card has been changed -- if the response codes were digits, then the *key card will give the digit corresponding to the right answers. (Also note: RES= is the same as Res=, which is the same as res=, which can even be the same as Responses=; Lertap really only looks at the first letter of the control words, and it doesn't care if letters are upper or lower case.)

Here are some other examples of valid Res=() declarations:

Res=(T,F)

(The subtest's *key card must contain Ts and Fs.)

Res=(A,B,C,D,E,F,G,H,I,J)

Res=(a,b,c,d)

(The subtest's *key card must contain lower-case letters.)

Res=(1,2,3,4,5,6)

(The subtest's *key card must have digits.)

Res=(A,B,C,D)

(Not required! This is the default setting for cognitive items.)

Critical note: the response codes seen in the Res= declaration tell Lertap what to look for when it reads the information in the Data worksheet's rows. If the response codes are upper-case letters, such as {A,B,C,D}, then Lertap will expect to find upper-case letters in the relevant columns of the Data worksheet. Nasty things can happen when, for example, the item responses seen in Data columns are lower-case letters, such as {a,b,c,d}, and the *sub card has Res=(A,B,C,D). This is a mis-match. Res=(A,B,C,D) tells Lertap to look for upper-case letters, but none will be found. Things will come a-crashing. (There's a bit more on this towards the end of the CCs sheet topic.)

Example C6:

Here's one more example of the *sub card in action:

*col (c2-c6)

*sub Title=(NewsQuiz), CFC, Wt=.5

*key ACCDB

CFC means "correction for chance", another scoring option entertained by Lertap. This control word isn't used all that often; it usually results in penalising students if they appear to be guessing (see Chapter 10 of the manual for more discussion). The Wt= declaration applies when the CCs worksheet defines more than one subtest, that is, when there are two or more *col cards. In this case, Lertap will usually generate a total test score by summing the subtest scores; the Wt= assignment controls how this is done. If Wt=0 then the subtest will not be included in the total test score.

Now we will exemplify the use of the other control cards for cognitive tests.

Example C7:

We'll add an *alt card:

*col (c2-c6)

*sub Name=(Followup TV9 news quiz), Title=(NewsQuiz)

*key ACCDB

*alt CDDDC

The *alt card is optional. Here it's telling Lertap that the last response code used by the first and last items is C, whereas the last response code used by all other items is D. Since there is no explicit Res= declaration, Lertap assumes Res=(A,B,C,D). (Note that this format of the *alt card differs from that shown in the manual. It's a new format, introduced in February, 2005. In the old format this *alt card would have been *alt 34443.)

Example C8:

We'll use a *wts card:

*col (c2-c6)

*sub Name=(Followup TV9 news quiz), Title=(NewsQuiz)

*key ACCDB

*wts 31121

The *wts card is optional. It indicates the number of points to be given for the correct answer, and it's only required when some of the items are worth more than one point. In this example, the correct answer to the first item, A, is worth 3 points, while the correct answer for the fourth item, D, is worth 2 points. All other items are worth one point.

If an item is to be worth more than 9 points, a *mws card has to be used. *mws cards are mentioned below.

Example C9:

Both *alt and *wts cards in use:

*col (c2-c6)

*sub Name=(Followup TV9 news quiz), Title=(NewsQuiz)

*key ACCDB

*alt CDDDC

*wts 31121

You understand this one, don't you? The right answer to the first item is A. It (the first item) uses three response codes, (A,B,C). A correct answer on the first item is worth 3 points.

Q: if I answer D on the fourth item, how many points do I get? Two.

What's the maximum score I can get over these five items? Eight.

If I answer C on the last item, how many points do I get? None; the right answer is B.

If I don't answer the third item, what happens? I get sent home early with instructions to have extra peanuts with my beer. (In truth: nothing. A non-answer to a cognitive item usually gets "scored" as a zero.)

Example C10:

Using Lertap's Big Gun, the *mws card:

*col (c2-c6)

*sub Name=(Followup TV9 news quiz), Title=(NewsQuiz)

*key ACCDB

*mws c2, 1, 0, 0, *

*mws c3, 0, 0, 1, 0

*mws c4, 0, 0, 1, 0

*mws c5, 0, 0, 0, 1

*mws c6, 0, 1, 0, *

This example is really the same as Example 9. We want to ease you into the idea of *mws cards by starting with an "easy" example.

Keep in mind that the default Res=(A,B,C,D) applies to this example, there being nothing to the contrary on the *sub card.

The *mws c2 card refers to the item whose responses are found in column 2 of the Data worksheet. This is, of course, the first item. Of the four potentially-possible responses to this item, (A,B,C,D), the *mws c2 card says that the first response is to get one point; the second and third responses are to get zero points, and the fourth response is in fact not used by this item -- hence the asterisk.

Look at the *mws cards above. They have the same format: the column number of the item in question, followed by the number of points corresponding to each of the item's response codes. If the item does not use one or more of the response codes, an asterisk is used.

Example C11:

More about the *mws card:

*col (c2-c6)

*sub Name=(Followup TV9 news quiz), Title=(NewsQuiz)

*key ACCDB

*mws c2, 1, 0, 0, *

*mws c6, 0, 1, 0, *

This example is the same as the last one.

We hear you saying "No it's not, come on now! The last example used five *mws cards; now there are only two".

Sure. You're right. What we should say is that this example accomplishes the same item scoring as the last example. Look at the three cards we've eliminated:

*mws c3, 0, 0, 1, 0

*mws c4, 0, 0, 1, 0

*mws c5, 0, 0, 0, 1

These cards say that the items found in columns 3, 4, and 5 of the Data sheet use all four response codes, have one correct answer, and award one point for the correct answer.

But this is the default. Lertap assumes all items will use all response codes, have one correct answer, and will award one point when the correct answer is selected. There's no need for *mws cards for these items -- their scoring is standard stuff.

So, what's special about the items in c2 and c6? They don't use one of the response codes. Now, this really isn't a big deal. Lertap would process the c2 and c6 items even if we didn't mention the fact that these items use just three response codes; Lertap's various reports would simply show that the fourth option, with a response code of "D" in this case, was not selected by anyone, and the Stats1b report would flag "D" as a poorly-performing distractor. Such things as test scores and coefficient alpha will not be not affected.

But why not do the job right? Lertap allows items to have a different number of options. The *alt card and the *mws card both allow you to set the record right, to inform Lertap that some items do not use all of the subtest's response codes. Use these cards and Lertap's reports will look a bit cleaner.

Example C12:

Still more about the *mws card:

*col (c2-c6)

*sub Name=(Followup TV9 news quiz), Title=(NewsQuiz)

*key ACCDB

*alt CDDDC

*mws c4, 0.5, 0, 0.5, 0

This example is quite typical. The item whose responses are coded in column 4 of the Data sheet is now being double-keyed. If someone selects the first response they get half a point. And, if someone selects the third response, they also get half a point. There are two "right" answers, each worth half a point.

*mws c4, 1, 0, 1, 0

Again there are two right answers, but now they're each worth one point.

*mws c4, 0.50, -0.50, 2.00, -0.75

Things are getting real fancy now. The best answer is the third one, for which a whopping two points are awarded. The first answer is worth half a point. The second and fourth answers now have negative scoring weights; a person selecting the second option loses half a point, whereas someone going for the fourth option will lose three-quarters of a point.

Example C13:

Some buildings do not have a 13th floor, and we don't have a 13th example, either.

Example C14:

Re-scoring all items at once:

*col (c2-c6)

*sub Name=(Followup TV9 news quiz), Title=(NewsQuiz)

*key ACCDB

*mws call, 1, 0, 1, 0

The items found in all of the subtest's columns are to be scored with one point for the first and third responses, with no points for the second and fourth responses. This sort of scoring is not at all common for cognitive items, not at all -- but if you want to do it, you can.

In the world of Lertap, *mws cards are the most potent cards going. They're dynamite. They completely override whatever information has come on preceding CCs cards.

A special form of the *mws card may be used when it's desired to quickly remove an item from a subtest. Click here to read about it.

In Lertap Version 5.25, another special form of the *mws card was introduced. It has this form:

*mws c12, 0, 1, 0, 1, other=1

To give credit to everyone for an item, even if they didn't answer the item, a card such as the following might be used:

*mws c12, 1, 1, 1, 1, other=1

The card above gives one point for each of the item's permitted answers, and it even gives people one point if they didn't answer the item.

Click here to read more about "other".

There are indeed times when, as in Example 10, a *mws card is used for each item. It may be only 2% of the Lertap-using world which will have an example of this sort, but it does happen -- we've seen it. In such a case, does the *key card make sense? No. But Lertap requires each and every cognitive subtest to have a *key card, so put one in (please).

Keep in mind that the manual has three chapters on CCs cards. Between what's written there, and what's appeared in this help topic, we hope you'll have an adequate to good grasp of CCs cards. But drop us a note if you've got questions: support@lertap.com.


Affective CCs

Affective control cards

This is the all-you-ever-wanted-to-know page about affective control "cards". First, some terms:

Let's say you had a couple of Likert-style items like these 'uns:

1) West Australian beaches are unsurpassed in the whole world.

1 Strongly disagree

2 Disagree

3 Undecided

4 Agree

5 Strongly agree

2) The beaches of Maui are better than West Australia's.

1 Strongly disagree

2 Disagree

3 Undecided

4 Agree

5 Strongly agree

Both of these items have five possible responses, or options, or alternatives, and use response codes of {1,2,3,4,5}. It's possible to have items with more options; Lertap allows up to 10 options per item. Items do not have to use the Likert style. The response codes used to not have to be digits (examples below).

Likert-style items are very common; another popular style is the semantic differential.

If we at Lertap central wanted to "score" these items, we'd be content to follow the conventional pattern of letting "strongly disagree" equal one point, "disagree" two points, ..., and "strongly agree" five points.

Respondents could end up with a low total "score" of two (2) points, and a top total "score" of ten (10) points. They'd get the low "score" if they answered "strongly disagree" on both items. They'd get the top "score" if they chose "strongly agree" on both items.

It's not really necessary to think about "scores" such as these, but some people find them useful. And, if they do, and if they work for WATC, the West Australian Tourist Commission, they'd likely want to reverse the scoring for the second item.

Say what? Reverse the scoring? You bet; it's a common happenstance. The WATC mob would want people to strongly disagree with the second item above. They'd give "strongly disagree" a "score", or weight, of 5, "disagree" a weight of 4, ..., and "strongly agree" a weight of 1. This way the top scores will come from people who not only love WA's beaches, but think they're better than those found around Maui.

Okay then, we've got some basic terms under the belt. Of course, if we're on one of those beaches, we may not have a belt to put them under, but let's proceed anyway.

Say we had five items of the sort shown above. Say we asked 200 WA-based people to respond to the items, and took the trouble to fly to Hawaii, paddle out to Maui, and ask another 200 folks to respond to the same five questions. We entered the 400 responses into a Lertap Data worksheet, with some sort of ID code in column 1, a location code of "W" or "M" in column 2, and the answers to the five questions in columns 3, 4, 5, 6, and 7. Were we to look down columns 3 through 7, we'd see 1s, and 2s, and 3s, and 4s, and 5s, corresponding to answers of "strongly disagree" through to "strongly agree".

Having entered the data, we need to go to work in Lertap's CCs worksheet.

Example A1:

We started with this set of two CCs cards:

*col (c3-c7)

*sub AFF

These two simple cards are all that some people might use to process the results. The *col card gets Lertap and Excel to read information from the Data worksheet, looking at five columns, 3 through 7. The *sub card has the AFF control word -- this is necessary in order to get Lertap to process the items as affective ones; without this card, and without the AFF control word on it, Lertap would have tried to process the items as cognitive ones, and would have wanted to find a *key card with the right answers to each item.

Lertap will "score" each item, giving a weight of 1 every time it encounters a 1 in the item's Data column, a weight of 2 for a response of 2, and so on. The minimum score on any item is 1, the maximum is 5.

Lertap will also make a "subtest score", or "scale score", for each respondent, placing these in its Scores worksheet. Such scores are just the sum of the item scores. Since there are five items, the minimum possible subtest score would be 5; the maximum would be what? Yes, 25.

Not everyone is interested in these scores. Many people are, but some aren't. Lertap makes them, and you'll just ignore them if you don't like to score affective items.

Example A2:

Example A1 was too simple if we're interested in scores. You've already heard that the second item was to have reverse scoring; you didn't know it, but the third item was also to be reversed. We have need for a *pol card:

*col (c3-c7)

*sub AFF

*pol +--++

The *pol card is usually optional for affective subtests. It's hauled out when some of the items are to be reverse-scored.

The *pol card above tells Lertap that the second and third items are to be reverse-scored -- that's what the minus (-) signs mean. The little plus (+) signs tell Lertap to score items 1, 4, and 5 in the usual, "forward", manner.

Example A3:

Now we'll put in some labels to grace Lertap's reports:

*col (c3-c7)

*sub AFF, Name=(Beach survey 1), Title=(Beachin)

*pol +--++

The Name shown above will appear at the top of some of Lertap's reports, such as Stats1f and Stats1b, while the Title will show up at the top of one of the Scores worksheet's columns. The Name can be any length; Title should be kept to 8 characters or less. If Name and Title are not given, Lertap defaults to Name=(Test 1), Title=(Test1).

Example A4:

Next we'll add some more control words to the *sub card, and then explain what they accomplish:

*col (c3-c7)

*sub AFF, Title=(Beachin), PER, SCALE

*pol +--++

The PER control word prompts Lertap to compute a percentage score for each respondent, being his or her score expressed as a percentage of the maximum possible score. For example, on our little test of five questions, the maximum possible score was 25; someone with a score of 15 would get a PER score of 60%.

SCALE gets Lertap to "normalise" the scores. It divides each person's score by the number of items. This is best used when all the items have the same number of options; it results in a score scaled back to the scores used at the item level.

For example, let's say someone scored 10 on our 5-item test. Divide this score, 10, by the number of items, 5, and SCALE=2.00 for this person. This might make us think that the person's "average" response to our items was "disagree", since "disagree" had a scoring weight of 2.

SCALE can be handy when processing an affective instrument with numerous subscales, each scale having a different number of items. If SCALE is used on each respective *sub card, then we can scan any person's SCALEd scores and quickly see their "average" positions on the 1-to-5 scale used to score each item. (The manual has a real-life example of just such a situation: see Chapter 8, where the MSLQ instrument is discussed.)

Example A5:

Now we will toss in two more control words for the *sub card:

*col (c3-c7)

*sub AFF, MDO, Title=(Beachin), Wt=0

*pol +--++

The MDO control word will get Lertap to turn off its missing data option. Lertap makes a standard response substitution when people don't answer an item: it gives them an item score equal to the mean of the item weights (note). In our example, the item weights range from 1 to 5; the mean is 3. A person not answering an item gets a score of 3; someone declining to answer all items would get a score of 15 on our 5-item scale.

To turn off this automatic substitution, use MDO on the *sub card, as exemplified above. To have more control over how missing data are processed, use *mws cards with an "other=" declaration, as mentioned below, under Example A11.

Revision note 12 September 2004: the way the MDO option works has changed. Now MDO effectively means "missing data out"; when MDO is present, subtest statistics are adjusted so that they exclude people who haven't answered items. This is further discussed in the following topic.

The Wt=0 declaration shown above says to Lertap: "As you go about summing all the subtest scores to make a total score, give this subtest a weight of zilch (zero)."

What's that, you say?

Well, whenever a CCs worksheet has more than one *col card, we're into a situation where there are multiple subtests. We are? Sure: each *col card defines a new subtest.

Granted, there aren't multiple *col cards in this example, but pretend there were.

Lertap's standard modus operandi is to add all the subtest scores together, making a total score for each person. Usually this total is just the sum of the subtest scores, much as if Wt=1 had been used on each subtest's *sub card. The Wt= declaration gives you the ability to control the way a subtest adds to the total score. Since Wts can be negative, you can even get a subtest's result to be subtracted from the scene.

How you doing? This is all pretty straightforward, is it not? We'll step up the tempo a bit, and get into some more advanced matters.

Example A6:

We've used an Res= declaration below, have a look:

*col (c3-c7)

*sub AFF, Res=(A,B,C,D,E), Title=(Beachin)

*pol +--++

Lertap assumes that affective items have five options, with the response codes corresponding to the options being (1,2,3,4,5). The Res= declaration on a *sub card tells Lertap this, but, when Res=(1,2,3,4,5), there's no need to explicitly say so. Lertap assumes Res=(1,2,3,4,5) for affective items. (For cognitive items, Lertap assumes Res=(A,B,C,D).)

If this isn't the case, an Res= declaration is required. In this example, the five affective items have five options, with the response codes for the items being (A,B,C,D,E). It's as if the items had this sort of format:

4) A few Emu Exports a day keep the doctor away.

A Strongly disagree

B Disagree

C Undecided

D Agree

E Strongly agree

If you had no trouble with this example, try the next one. (Sooner or later we'll stump you.)

Example A7:

Here's a big example, getting into the real intricacies of Lertap and item weighting:

*col (c3-c7)

*sub AFF, Res=(5,4,3,2,1), Title=(WAuni)

*pol +--++

Here we're still in a situation where the items have five options, and the response codes are back to (1,2,3,4,5). However, now a response of 5 is to get a weight of 1, a response of 4 a weight of 2, ..., a response of 1 a weight of 5.

It's as if the items looked like the one below, where the Likert scale has been re-ordered so that "strongly agree" is the first option (very common):

5) West Australian universities are truly world class.

1 Strongly agree

2 Agree

3 Undecided

4 Disagree

5 Strongly disagree

Those of us based in West Australia would want this item to be scored in a manner which gives the highest score to the first option. Ordinarily, the first option gets the lowest weight. We could use a *pol card to reverse this, and most Lertappers probably would (including us). However, we said we'd step up the tempo; we've started to wade into deeper water.

An understanding of this example gets into the very basics of Lertap's affective item scoring. The default scoring weight applied to any particular response code corresponds to the response code's ordinal position in the Res= declaration. If 1, or A, or 5 is the first response code to appear in Res=, it gets a scoring weight of 1 (but let's say 1.00 to emphasize that we're talking about a real number).

If Res=(1,2,3,4,5), scoring weights are 1.00, 2.00, 3.00, 4.00, and 5.00.

If Res=(A,B,C,D,E), scoring weights are 1.00, 2.00, 3.00, 4.00, and 5.00.

If Res=(5,4,3,2,1), scoring weights are 1.00, 2.00, 3.00, 4.00, and 5.00.

The default scoring weights have no correspondence to the actual response codes. They're based entirely on the ordinal position of the response codes in the Res= string. The entries in the Res= declaration are never read as numbers. Never -- they're just characters.

If Res=(5,6,7,8,9), scoring weights are 1.00, 2.00, 3.00, 4.00, and 5.00.

If Res=(1,2,3), scoring weights are 1.00, 2.00, and 3.00.

If Res=(w,x,y,z), scoring weights are 1.00, 2.00, 3.00, and 4.00.

Example A8:

The stage is set; we now introduce the most powerful card in Lertap's mighty arsenal: the *mws card.

Example A2 above looked like this:

*col (c3-c7)

*sub AFF

*pol +--++

We'll knock out the *pol card, and instead go with these CCs cards:

*col (c3-c7)

*sub AFF

*mws c4, 5.00, 4.00, 3.00, 2.00, 1.00

*mws c5, 5.00, 4.00, 3.00, 2.00, 1.00

The *mws cards explicitly apply scoring weights for the two items whose responses are found in columns 4 and 5 of the Data worksheet. They're saying that the first response code in the Res= declaration is to get a weight of 5.00, while the last response code is to get a weight of 1.00.

There isn't an Res= declaration on the *sub card, is there? No. In the absence of one, Lertap assumes Res=(1,2,3,4,5).

The two *mws cards have reversed the scoring for the second and third items, respectfully located in column 4 (c4) and column 5 (c5) of the Data worksheet.

Why aren't there *mws cards for the items in c3, c6, and c7? There really are; if you could look deep inside the heart of Lertap, you'd see that it's effected these statements:

*mws c3, 1.00, 2.00, 3.00, 4.00, 5.00

*mws c6, 1.00, 2.00, 3.00, 4.00, 5.00

*mws c7, 1.00, 2.00, 3.00, 4.00, 5.00

The weighting pattern seen in these three cards is the default pattern for items with five response codes. By "default" is meant "not requiring mention; this is what I'll do unless you tell me otherwise". This being the case, we don't need to say anything. Let Lertap apply its default weights for each item unless we say different. Use *mws cards to "say something different".

Now, in this case, we have used two *mws cards in place of one *pol card. That's not real efficiency. The majority of Lertap users are happy with the *pol card, but there are some advantages in using *mws cards. One advantage is that there's no ambiguity with *mws cards -- they make it absolutely clear how item responses are to be scored, and explicitly indicate the items affected.

About that idea of looking "deep into the heart of Lertap" ... a Lertap workbook's Sub worksheets provide quite a detailed glimpse of exactly how item weights have been set up. To read more about Sub sheets, just click here.

Example A9:

Here's an example which shows off a special use of the *mws card:

*col (c3-c7)

*sub AFF

*mws call, 5.00, 4.00, 3.00, 2.00, 1.00

'*mws call' means we have a multiple-weights specification which is to apply to all the items mentioned on the preceding *col card. This use of the *mws card often comes into play when there are items such as the following:

5) Mt Barker wines are as fine as those of California's Napa Valley.

1 Strongly agree

2 Agree

3 Undecided

4 Disagree

5 Strongly disagree

The *mws call card will reverse the default scoring weights, equating the first response code, 1, with 5.00 points, and the last response code, 5, with 1.00 points.

Astute readers might note that the scoring accomplished by the three CCs cards of this example is the same scoring achieved in example A7. In A7 we used a special Res= declaration to do the job, whereas now we're using *mws call.

Examples A7 and A9 will result in the same item scoring, but Lertap's Stats1f and Stats1b will differ. One could see the differences by running with the following set of CCs cards:

*col (c3-c7)

*sub AFF, Res=(5,4,3,2,1), Title=(WAuni)

*pol +--++

*col (c3-c7)

*sub AFF

*mws call, 5.00, 4.00, 3.00, 2.00, 1.00

We've combined the two examples, A7 and A9, making two subtests from the same items. Now the two full-statistics reports, Stats1f and Stats2f can be compared; item and test stats will be the same, but the order in which item response codes are listed will differ. The same will hold for the Stats1b and Stats2b reports.

Example A10:

Item scoring weights can be any real number, positive or negative:

*col (c3-c7)

*sub AFF, Res=(1,2,3,4,5,6,7), Title=(CapesAtt)

*mws call, -3.00, -2.00, -1.00, 0.00, 1.00, 2.00, 3.00

This type of weighting is sometimes seen when semantic differential items are used, as in the following example:

8) The weather in the southwest capes region of West Australia is:

rotten

__

__

__

__

__

__

__

wunderbar

There are seven blanks above. If someone clicks the fourth blank, the centre one, their response will be entered in the Data sheet as a 4, and the *mws card above will have their response scored as 0.00 points. Selecting the first blank will see a 1 entered on the Data sheet (because it's the first blank), and a weight of -3.00 points will apply.

How many points if someone ticks the sixth blank? 2.00.

Example A11:

Enter the *alt card:

*col (c3-c12)

*sub AFF, Res=(1,2,3,4,5,6,7), Title=(CapesAtt)

*alt 55555 77777

To try and explain this set of cards, we ask you to imagine that we've given a survey with ten items. The first five items are of the Likert style, and use response codes of (1,2,3,4,5). The last five items employ the semantic differential style, and use response codes of (1,2,3,4,5,6,7).

The *alt card tells Lertap this. It says that the first items use the first five response codes seen in the Res= declaration, while the last five items use seven of the response codes (which of course is all of them). It's still possible to add a *pol card in cases like this, as shown here:

*col (c3-c12)

*sub AFF, Res=(1,2,3,4,5,6,7), Title=(CapesAtt)

*alt 55555 77777

*pol +--++ +++++

But there's a probable problem here. Users of semantic differential items often like to have negative weights corresponding to the negative side of their questions, as seen above in Example A10. The set of four CCs cards above is not right; the five semantic differential items will have scoring weights of 1.00, 2.00, ...., 7.00. We need some *mws cards:

*col (c3-c12)

*sub AFF, Res=(1,2,3,4,5,6,7), Title=(CapesAtt)

*alt 55555 77777

*pol +--++ +++++

*mws c8,  -3.00, -2.00, -1.00, 0.00, 1.00, 2.00, 3.00

*mws c9,  -3.00, -2.00, -1.00, 0.00, 1.00, 2.00, 3.00

*mws c10, -3.00, -2.00, -1.00, 0.00, 1.00, 2.00, 3.00

*mws c11, -3.00, -2.00, -1.00, 0.00, 1.00, 2.00, 3.00

*mws c12, -3.00, -2.00, -1.00, 0.00, 1.00, 2.00, 3.00

Of course, we could have used *mws cards for all items, and, in this case, we'd then have:

*col (c3-c12)

*sub AFF, Res=(1,2,3,4,5,6,7), Title=(CapesAtt)

*mws c3,   1.00,  2.00,  3.00, 4.00, 5.00, *, *

*mws c4,   5.00,  4.00,  3.00, 2.00, 1.00, *, *

*mws c5,   5.00,  4.00,  3.00, 2.00, 1.00, *, *

*mws c6,   1.00,  2.00,  3.00, 4.00, 5.00, *, *

*mws c7,   1.00,  2.00,  3.00, 4.00, 5.00, *, *

*mws c8,  -3.00, -2.00, -1.00, 0.00, 1.00, 2.00, 3.00

*mws c9,  -3.00, -2.00, -1.00, 0.00, 1.00, 2.00, 3.00

*mws c10, -3.00, -2.00, -1.00, 0.00, 1.00, 2.00, 3.00

*mws c11, -3.00, -2.00, -1.00, 0.00, 1.00, 2.00, 3.00

*mws c12, -3.00, -2.00, -1.00, 0.00, 1.00, 2.00, 3.00

We know you can just about understand these cards, but realise you might want to know what the asterisks are doing on the *mws cards for the items in c3 to c7. The asterisks tell Lertap that the respective items do not use the last two response codes seen in the Res= declaration.

It is possible to use special forms of the *alt and *mws cards when it's required to quickly remove an item from a subtest. These forms can be very handy in certain circumstances; please refer to the 'Remove an item' topic for examples.

In Lertap Version 5.25, it became possible to use the "other=" control word on a *mws card, as exemplified below:

*mws call, 1, 2, 3, 4, 5, other=9

The line above says that anyone not selecting one of an item's five permitted responses will get a score of 9 points. The use of other= presents a more flexible way to handle missing affective data than does the MDO option discussed above.

For more about using the other= control word, and item weighting, please click here.

Example A12:

Just one more example, and this one's simple:

*tst c2=(W)

*col (c3-c7)

*sub AFF, Name=(Beach survey 1), Title=(Beachin)

*pol +--++

This example displays use of the *tst card. The *tst card is telling Lertap to make a new Lertap workbook, one whose Data worksheet will contain copies of only those Data records which have a "W" in their second column. We'd probably then set up another Lertap analysis with these cards:

*tst c2=(M)

*col (c3-c7)

*sub AFF, Name=(Beach survey 1), Title=(Beachin)

*pol +--++

Now Lertap is being told to make a new workbook whose Data records will be only those with an "M" in column 2. We have now set up two new data sets, two new Lertap workbooks, one for the respondents from West Australia, and one for those from Maui.

(For a more thorough discussion of using the *tst card, and breaking out subsets of records, see the section titled "A survey with multiple groups" in Chapter 8 of the manual.)

Good going; you stuck it out and have come to the end of this topic / page. Lertap provides extensive support for different item scoring methods, as we've exemplified above. Of course there's more material in the manual, where three chapters are devoted to a discussion of CCs cards.

Still, a few years of experience have shown us that the matter of item scoring in Lertap at times creates questions among users, no matter how much documentation we might provide. Should you have questions, whiz off a note to us: support@lertap.com.


Missing data

In Lertap, "missing data" is a term which usually means that a person has not answered, or has omitted, an item.

If you look at a Lertap Data worksheet and see empty cells, such cells usually indicate that there was no answer to the item corresponding to the cell. In the screen snapshot below, Virgo has not answered Item 2, nor Item 7; Westphal has left three items unanswered (2, 5, and 7); Xeno did not answer Item 2; Yalso has not answered Item 7.

There are quite a number of users who prefer to use a special code for the case of unanswered items. Instead of leaving a cell empty, they might use a 9 to indicate missing data (there are historical reasons for this; some of the most popular data analysis programs have traditionally used 9s to represent missing data). When using scanners to process mark-sense answer sheets, the software driving the scanner may have its own missing data code, such as an asterisk (*).

To fully understand how Lertap processes missing data, it helps to have a good understanding of what Lertap calls "response codes".

The letters (or digits) seen under a Lertap Data worksheet's item columns represent response codes. In the example above, it seems that items have used response codes of A, B, C, and D; this set, {A,B,C,D}, is, in fact, Lertap's default response code set for cognitive items -- for affective items, the default response code set is {1,2,3,4,5}.

It is common for users to have items which use other response codes. Whenever a test does not use the default response codes, the response codes used by that test's (or survey's) items are specified by using an Res= declaration on a *sub card.

For example, Res=(A,B,C,D,E,F), Res=(1,2,3,4,5,6,7), and Res=(t,f) are all valid Res= declarations. In Lertap, a test may use up to 26 response codes.

Now, with this understanding of response codes in hand, the definition of missing data can be made a bit more precise: in Lertap, an item response is said to be missing whenever a cell in the Data worksheet has an entry which does not match the response codes used by the item. When this happens, Lertap says it has encountered an "other" response.

As Lertap goes about tallying item responses, it keeps track of the number of "other" responses in a special bin.

What's this bin called? The "other bin", naturally. Deep inside Lertap, each and every item is assigned a storage bin for "other" responses. The contents of an item's "other" bin are displayed in many of Lertap's reports.

As an example, look at these snapshots:

In these two examples, the "other" bin for Q11 has n=6, meaning that there were six people missing data on the item. For Q35, there were n=9 people missing data.

So. Are you now full bottle on Lertap's definition of missing data? And an expert on Lertap's "other" bin?

Goodonyamate. But wait, there's more ....

What this stuff under the "wt." column? The answer to this question is also very relevant to understanding how Lertap processes missing data, especially for affective tests.

The "wt.", for "weight", indicates how many points are associated with each item option.

On Q10, the 28 people who chose response code A (usually referred to as option A, or alternative A) will get 1.00 points for their answer. Choosing any of Q10's other options gets no points. You guessed it: Q10 is a cognitive item whose right answer is A.

Not answering Q10 gets no points. Not answering Q10 gets no points. We have repeated this as it's important: an item's other bin can have scoring points attached to it.

Look at Q35, an affective item. Each and every response code has associated scoring points, and, should someone not answer Q35, they will still get 3.00 points (!).

What we've displayed in these two examples represents Lertap's default handling for cognitive and affective missing data. A cognitive item's default missing data scoring action is 0.00 points; an affective item's default missing data score is equal to the mean of the other scoring wts.

With this in mind, take a break. When you return, have a look at the following topics to read more about Lertap and missing data.


Did-not-see option

In March 2006 new options were added to the System worksheet which allow users to adjust cognitive item statistics so that they are based only on those cases (students) that had a chance to answer the item.

The did-not-see option is activated by placing "yes" in column 2 of the appropriate row in the System worksheet; in the example above, this is row 40.

Use of this option requires a corresponding did-not-see code, a single character recorded in the Data worksheet whenever a student did not have the chance to answer an item (for whatever reason). In the example above, the code is a blank. The code may be any single character, including a blank, an upper-case letter from the Latin alphabet (ABC...XYZ), a lower-case letter (abc...xyz), or a single Arabic numeral (0123456789).

How does Lertap process item responses with this new option? Well, for any item, Lertap first looks to see if a student's response corresponds to one of the response codes used by the item's options. If it does not, then Lertap looks to see if the "response" matches the did-not-see code, assuming the did-not-see option has been activated. If it does not, then the response is classed as "other", a response category often referred to in Lertap documentation as "missing data".

Lertap will automatically adjust all of its item statistics so that they exclude did-not-see cases (that is, of course, assuming the did-not-see option has been activated -- if the option has not been activated, then what would have been a did-not-see response will be included in the "other" response category, and processed as missing data).

Use the Freqs worksheet to check on missing data and did-not-see cases, and note: if the did-not-see code is a blank, Freqs will include did-not-see cases in its "other" line.

To see what the various "Stats" reports look like when the did-not-see option is on, just page forward to the following topics.

Scoring

Whenever the did-not-see option is in effect, users have the option to have Lertap calculate two scores for every person on each subtest defined in the CCs worksheet: the usual subtest score, and a percentage score which is based on the maximum possible score which could have been obtained on the items actually presented.

The first score, the "usual" subtest score, will be the number of points earned on the number of items the person had the opportunity to answer. In the cognitive case, the most common scoring method is to give one point for each correct answer -- in this case, the "first score" will simply be the number of items right.

The second score is only produced when "yes" is found in the System worksheet in the second column of the line which says "Create an adjusted percentage score?". This score is formed by dividing the first score, the "usual" subtest score, by the maximum possible score which the person could have had if s/he had correctly answered every presented item; this figure is then multiplied by 100 to get a percentage index.

Coming back to the common cognitive case, with one point for each correct answer, the second score is the number of answers right divided by the number of items presented, multiplied by 100.

Note that both of these scores exclude items which the person did not see, that is, did not have the opportunity to answer.

It is not necessary to use the PER control word on the *sub card in order to get the second score: it's computed automatically, providing "yes" is found in the second column of the line which says "Create an adjusted percentage score?".

Adjusted?

When would you want to have the percentage scores "adjusted"? What does "adjusted" mean, anyway?

Well, it turns out that certain online test generators can create tests of variable length; some students might see a 48-item test, others a 45-item test, and still others a 50-item test. A student's test score will depend on the number of items s/he had the chance to answer. Two students with the same "usual" subtest score, say 35, will not have the same percentage score if they have been presented with a different number of items.

For example, if Jorge got a score of 35 on a test with 50 items, his percentage score would be 70%, assuming one point for each correct item. Suppose it turned out that Marisol also got a score of 35, but she was presented with only 45 items; her rounded percentage score will be 78%.

When you've asked Lertap to "Create an adjusted percentage score", the percentage test score will be adjusted according to the items a student actually saw.

If all students get the same number of items, there is no need to create an adjusted percentage score. Ah, but wait a minute! We should adjust this comment: if all students are presented with the same number of items, an adjustment is not required if all items are scored the same way. When items are scored in different ways, then the adjustment might again be useful. For example, if Item 26 is worth one point, but Item 27 is worth two points, then we might well want to make an appropriate adjustment to scores, depending not only on how many items a student saw, but also on the number of points the items are worth.

Lertap creates the adjusted score by dividing the "usual" subtest score by the actual maximum possible score a student could have achieved, based on the items presented to the student.

Sample Scores output

As you page forward to following topics, you'll come across an example from an actual online testing situation, one which used the Test Pilot system from McGraw-Hill.

Over 400 students at a large North American university took an online version of a test delivered by Test Pilot. Sampling from a pool of 80 cognitive multiple-choice items, Test Pilot served up tests whose lengths varied: some students were presented with 40 items, some with 43, some 45, and others 48. (Please read the second "tidbit" below -- the situation was actually a bit more complicated.)

The Lertap Data worksheet turned out to have 93 columns for item responses. Why 93 when there were only 80 items in the pool? Because a few items allowed for multiple responses (see second tidbit below).

In this example, blanks were used as the did-not-see code. Were you to look at the item responses for any student, scanning from left to right over the 93 columns allocated, you'd see actual responses, many blanks, and, for those items not seen by a student but not answered, a 9.

We could, and will, entertain a variety of Lertap scoring "methods" which will demonstrate how the did-not-see options affect test scores.

To begin, say we have the did-not-see option off, that is, we have "no" in the System worksheet row which says "Use a did-not-see code?". Also, assume we have the PER control word on the *sub card.

The Scores output will look as follows:

The first student had a test score of 38. Lertap says that the corresponding percentage score is 40.9, a value found by dividing the score, 38, by 93, the maximum possible test score if each item is scored giving one point for a correct answer.

But this is not correct in this case; no student had the opportunity to answer 93 items -- Test Pilot gave each student a random sample of approximately 40 items.

So, we activate the did-not-see option by putting "yes" in the second column of the System worksheet row which says "Use a did-not-see code?". We have "no" in the "Create an adjusted percentage score?" row. And, we still have PER on the *sub card. Our Scores now look as follows:

See how the percentage scores have changed? Lertap is now basing its percentage score on the maximum possible score which could have been earned on the actual items presented to the student. For the first student, the maximum was 48.

Dividing 38 by 48 and multiplying by 100 gives the 79.2 seen as the EE101% score for the first student.

Now, for the second student, does 30 divided by 48 equal 62.5? Yes. The problem is that the second student was presented with 40 items, not 48. The percentage score is wrong.

Whenever students are presented with a different number of items, or whenever items in the pool have different scoring patterns, "yes" should be used in the "Create an adjusted percentage score?" row.

So, let's see what happens. We put in that "yes". Do we still have PER on the *sub card? No, now it's not needed -- there would be no harm in having it there, but it is not required. Here are the scores:

The percentage scores above have each been adjusted, according to the maximum possible score a student could have obtained on the set of items s/he was presented with.

Yes, Miss? (A young woman in the fortieth row of the third balcony has her hand up.)

Why wouldn't I just use the adjusted percentage score option all of the time? Why not put that yes where it's supposed to be, and just leave it there for always?

A good question, thank you. The answer: Lertap pinches extra memory from the computer whenever it has to keep track of the maximum possible score each student could have achieved given the items presented. You save memory space, and also a tiny bit of processing time, by not adjusting the percentages. If you know that each student was presented with the same number of items, and all items were scored the same way, then say "no" to the "Create an adjusted percentage score?" option. Now you know, no?

Related tidbits:

Several learning management systems support the development and delivery of online cognitive and affective tests. Some, such as Angel Learning, can be made to randomly sample items from a database, presenting different students with different versions of a test, each with the same number of items. In classical test theory, when certain conditions have been met, these versions might be termed parallel forms, or equivalent forms. However, as of March 2006, the data file of item responses created by Angel does not indicate which test "form" a student took. Fortunately, the Angel output is padded with did-not-see codes so that each data record has the same length, a length equal to the total number of items in the database.

In the Test Pilot example discussed above, the test included a few items which used a "Check all of the following options which would be correct" format, effectively turning a single item into multiple true-false items. Test Pilot actually served up the same number of items to each student (40), but those items which used the "check all of the following" format did not consistently offer the same number of options. The practical end result was as described above: students received a variable number of test items.

Did-not-see data will affect the calculation of alpha, Lertap's reliability coefficient -- alpha is NOT corrected for did-not-see cases.


MDO cognitive, Statsf

MDO for cognitive tests (Statsf)

 Return to Introduction

There's a control word, "MDO", which may be used on the *sub card to control how Lertap processes missing data. MDO may be used with both types of test, cognitive and affective. The letters stand for "missing data out", meaning that records with missing data are to be excluded from Lertap's various calculations.

This topic discusses the effect of the MDO and did-not-see options on the cognitive test reports produced by Lertap; a following topic does likewise for affective tests.

Look at these CCs lines for a cognitive test:

These eight "cards" involve the same 25 items. Being the avid reader you are, you know you've seen these items before -- they're from the Lertap quiz.

The cards define two subtests. The only difference between the two is that the second one has the MDO option on; you can see it on the 6th line.

Statsf reports

A squiz of the Lertap's Stats1f and Stats2f reports will serve to highlight the effect of using MDO:

The report above comes from Stats1f. It summarizes the performance of two items, Q15 and Q16, using a variety of statistics.

Both items use four options, employing response codes {A,B,C,D}.

A student selecting option B will get 1.00 points towards her/his test score, as seen under the wt. column -- we'd conclude that both items have just one correct answer as all the other wt. values are zilch (zero).

Thirty (30) students got Q15 right. Sixty (60) students were involved, so p for Q15's option B is 0.50, which is, of course, 30 divided by 60.

The pb(r) and b(r) columns are, respectively, point-biserial and biserial correlation coefficients, indexing the relationship between option selection and the criterion score. If the item forms part of the criterion, as it does in this case, Lertap applies a part-whole correction, taking out the inflation the correlation coefficients would otherwise have (see the manual for a more extensive discussion).

The avg. column shows the average criterion score for those students selecting each item option. On Q16, the 36 students who selected option B had average criterion scores of 15.53. As a z-score, 15.53 is 0.42 (the manual has more to say; you ought to read it some day, perhaps when next at the beach).

Note that Q16 has an "other" line. Three students had missing data for Q16. They were weak students; their avg. was low, as it was for those who chose distractors C and D.

Okay? Got it? Good; now have a peep at the stats for the same two items after the MDO option has been used:

There's no change in the stats for Q15; all 60 students answered that item. But Q16 has changed; many of Q16's stats above, from p through z, differ from the Q16 stats seen earlier. Why? Because the three students missing an answer to Q16 have been excluded from the calculation of the stats.

Look at the p column, for example. For Q16's option B, p is now 0.63, corresponding to 36 divided by 57, not 36 divided by 60.

The avg. values are now computed using just the criterion scores for the 57 students who answered Q16. This applies to the other line too: 5.67, as a z-score in the distribution of 57 scores, would be -1.06.

Now, take a few seconds and sum down the p column for Q16. In the first report above the sum is 1.00 (100%). But in the second report the sum comes to 1.05 (105%). What's up?

The 0.05 p value for Q16's "other" row is just indicating the proportion of respondents who did not answer the question. The other four p values for Q16 sum to 1.00, which is what is expected when the MDO option is in use.

Statsf reports and the did-not-see option

When you've got the did-not-see option going, the Statsf report will be quite similar to what you've just seen immediately above. Look:

There's just a wee difference in these results. Can you spot it? The other line is now missing most of its stats.

What about the joint operation of MDO and did-not-see. What happens then? Got time to look at results for two items delivered to 421 students over the internet, using the Test Pilot system from McGraw-Hill?

In this example, a blank was used as the did-not-see code, and 9 as the code for missing data.

On Q50, three students failed to provide an answer, while 106 students did not see it (Test Pilot presented Q50 to 315 of the 421 students). Q80 was presented to 320 students, and they all answered it. Look at the corresponding Statsf summaries:

The statistics for both Q50 and Q80 have been computed by excluding the students who did not see the items, and by also excluding those with missing data.

Were you to sum the p values for Q50's four options, you'd have 0.37 + 0.30 + 0.20 + 0.13, or 1.00 (100%). The 0.26 on Q50's other line is the proportion of students who did not have an answer for the item, either because they did not see the item, or saw the item but did not answer it.

Let your mouse hover over the 0.26 value, and behold:

See how it works? When Lertap is running with both options, did-not-see and MDO, then the other line will have stats only when there were some students who did not answer the item. For Q50, the avg. criterion score for the three students who were missing data was 23.67; when this avg. score is inserted into the distribution of criterion scores for those 312 students who did answer the item, the corresponding z-score is -0.99.

Related tidbit:

As discussed in the manual, when more than one option to a cognitive item has a non-zero "wt." value, the pb(r) and b(r) statistics are corrected for part-whole inflation only for the option having the greatest wt.


MDO cognitive, Statsb

MDO for cognitive tests (Statsb)

 Return to Introduction

The previous topic discussed how the MDO and did-not-see options affect the information reported in Statsf reports.

Now you're set to see the corresponding Statsb reports:

The first report has an "other" column which indicates the number of people with missing data on each item, expressed as a percentage figure. The statistics in the diff. and disc. columns are based on calculations which include the people with missing data.

In the second report, the "other" column has been replaced by the "n" column. The entries in this column indicate how many people answered each item, and the various percentage figures, plus the diff. and disc. values, are based on n, that is, they exclude missing data. Lertap uses what is termed a "pairwise" exclusion rule to calculate the diff. and disc. values: if a student is missing data for the item, or did not see it, s/he is excluded from the calcs. for that item.

The report above corresponds to the Test Pilot results mentioned in the previous topic. The numbers seen in the "n" column exclude the number of cases with missing data (if any), as well as the number of cases who did not see the item (if any). To see how many cases were missing data, or did not see the item, refer to the Statsf report which corresponds, or to the Freqs report.

Related tidbit:

A reminder from Chapter 10 of the manual: the Statsb reports are computed on an item level. The disc value they display is a conventional product-moment correlation between the item and the criterion, corrected for part-whole inflation.


MDO cognitive, Statsul

MDO for cognitive tests (Statsul)

 Return to Introduction

The two preceding topics have discussed how the MDO and did-not-see options affect Statsf and Statsb reports. Now: Lertap's third report for cognitive items, Statsul -- what happens when the MDO and did-not-see options are used?

First up, above, the standard format for Statsul, what's seen before the MDO and did-not-see options are put to use.

Item Q50 is from the Test Pilot system mentioned in the previous topics. Over 400 students took the "EE 101 T3" exam over the internet; 25.2% of the students did not see Q50 as Test Pilot did not present it to them, while just under 1% of the students who saw Q50 didn't answer it.

The table above gives Q50 results after the MDO option has been turned on. Its statistics are based only on those students who answered the item. This means that all the did-not-see people, plus the we-saw-it-but-did-not-answer-it people, have been excluded from the proportions and from the calculations underlying U-L diff. and disc. The did-not-sees are excluded at this point as the did-not-see option has not yet been used; the did-not-see code has been processed as missing data.

In this table (above), the did-not-see option has been activated, but MDO has not. The "other" column is now indicating the proportion of students in each group, from upper down to lower, who were presented with Q50 by Test Pilot, but did not answer it. Those not presented with Q50, the did-not-sees, have been excluded.

For a refresher on how the U-L diff. and U-L disc. values are calculated, pay a visit to your local ice cream shoppe, then have a look at Chapter 10 of the manual.


MDO affective

MDO for affective tests

The "MDO" control word is used on the *sub card to get Lertap to exclude cases with missing data from its calculations. MDO may be used with both types of test, cognitive and affective.

The discussion found in this topic assumes some familiarity with material found in the topics immediately preceding. If you haven't been through them, take a few minutes to read the "Missing data" topic, followed by the topic dealing with the "Did-not-see option". Then report back here.

Ready, set, go? Have a look at the following CCs lines:

The lines above set out two subtests. Both are affective as the "Aff" control word has been used on each of the *sub cards.

Both subtests involve the same ten items; no doubt you recognize the subtest? Right -- it's the set of Likert-style "Comfort" questions found on the Lertap quiz. And, no doubt you also recall that the items themselves may be see in Appendix A of that best-seller, the Lertap manual? Very good.

The only differences between the two subtests are found in the *sub lines. The second subtest uses the "MDO" control word.

Okay; rig yourself up with a refreshment of some sort, polish your glasses, and have a gander at Lertap's reports for these two subtests.

The Statsf reports

The first table above shows item stats for Q27 and Q28 without MDO, while the second table reflects the results of using MDO.

Q27's stats are the same in both tables, are they not? No-one omitted this item, so the statistics are unchanged, aren't they?

No. In fact, they're not unchanged (fooled you, eh?). Everything's the same until we get to the pb(r), avg., and z columns, wherein some changes enter.

To understand why Q27's results differ, look at the "other" row for Q28.

In the first subtest, without MDO, Lertap has wt.=3.00, giving 3.00 points to the two (2) people who did not answer Q28. Not so in the second subtest, where those two people have been stripped of scoring points. There are two different scoring methods in operation here: without MDO, people missing an answer to an item are given points equal to the average value of the wt. figures for the item's options. When MDO is active, as in the second subtest, no points are given when someone omits an item.

The result? The subtest scores will differ. Scores on the first subtest will be higher as people who miss out items are still getting points. The mean (average) of the subtest scores on the first subtest will be higher than that for the second subtest; the point-biserial correlation values, pb(r), between an option and the criterion score, the subtest score, are likely to differ, as are the avg. and z values. The more missing data, the greater these differences are likely to be.

Even though everyone answered item Q27, the criterion measure used to calculate item option statistics, pb(r), avg., and z, differs from the first subtest to the second, generally resulting in different values for item Q27's output.

Now, about Q28. As noted, two people did not answer this question. Compare the values found in the % column for Q28: they're greater in the second subtest. The % figures for Q28 in the second subtest, the one using MDO, have been calculated with n=58, the number of people who actually answered the item. In the first subtest, the % values were calculated with n=60, the total number of people taking the test (survey).

In addition, the pb(r) values for the item options seen in the second subtest have been calculated on a pairwise basis -- they are based only on the people who actually answered the item.

To read a bit more about how Lertap computes the Statsf figures, go back for a look at the "MDO cognitive, Statsf" topic.

There are two main differences between the Statsf reports for cognitive and affective subtests: it is rare for people omitting a cognitive item to get scoring points, so the statistics for cognitive item options may not be noticeably different going from no MDO to MDO. Lertap will apply a correction for inflation to the pb(r) and b(r) values corresponding to the right answer to a cognitive question, but this correction is not applied in the Statsf report for affective items (but it is for Statsb: see below).

The Statsb reports

In the normal case, without the MDO option, the Statsb report for cognitive items has an "other" column which indicates the percentage of non responses to an item. This column changes to "n" when MDO is in operation, as may be seen above.

Q27 has no missing data; its mean and s.d. values are the same in both tables. The Q27 cor. figures differ for the reason found earlier in the Statsf reports: the values of the criterion measure, the subtest score, change as we go from no MDO to MDO.

Q28's figures differ almost everywhere. The percentages, mean, s.d., and cor. statistics for the second table, where MDO is having its impact, are all computed using only the responses from the 58 folks who actually answered this item. In some other data analysis systems, such as SPSS, the correlation (cor.) between Q28 and the criterion would be said to done on a pairwise basis: only when a person has data for both variables are that person's results used in the calulations.

Another note about the cor. values found in the Statsb reports: they are corrected for part-whole inflation.

Using the did-not-see option

Suppose the did-not-see option has been turned on, with X used as the did-not-see code. The respective lines in the System worksheet would look like this:

Next, have a look at a snippet of Freqs output:

Six people did not see Q35; three did not answer it.

Okay? Now, suppose MDO is not operating. The Stats1f output for Q35 will look like this:

If the MDO option is turned on, the output will change:

To grasp what Lertap has done, look down the % column for these two reports.

In the top report, the % values (and the columns to the right, from pb(r) to z) are based on n=54; the six people who did not see Q35 have been excluded from the calculations.

In the next report we've got MDO operating, and now we'll have n= what? Fifty-one (51). In this case, the report excludes the six did-not-sees, and the three did-not-answers.

Right. What about the corresponding Statsb reports? Thought you wouldn't ask. Here they be:

Where do you stand now? You see what happens, or are you in the did-not-see group?

Questions? Crank up your email program, and point it at: support@lertap.com.


Pre-scored items

It is usually the case that the entries found in the columns of the Data worksheet correspond to the response codes selected by each person.

For example, have a squiz at this snippet from rows 2 and 3 of a Data worksheet:

On Q20, this person selected the response which had been coded as A. Apparently s/he did not answer Q21. On Q22, the person selected the response coded as E.

The column entries change to digits from Q26 on, but the meaning is probably the same: on Q26 the person selected the response coded as 1 (one), while on the next item, Q27, s/he chose the response which had been coded as 3.

What we're looking at here is part of the Lertap Quiz data set, described in Appendix A of the manual. This quiz consisted of 25 cognitive items, Q1-Q25, followed by 10 affective items, Q26-Q35. The cognitive items used letters as response codes, while the affective items were of the Likert style, with 1 the code for "strongly agree", and 5 the code for "strongly disagree".

How many points did this person get for her/his answer of A on Q20? We don't know; we can't tell just by looking at the data above. And, even though there are digits in some of the columns, we can't assume that a "1" for Q28 means that the person got one point for his/her answer.

These item responses have yet to be scored.

Okay? Consider now another case. Suppose a cognitive test included the following question:

33) Read the five sentences below, and place a tick next to those sentences which use the pluperfect tense.

Student answers to a question like this one have to be scored by hand. How? Well, if there were three pluperfect sentences, and the student found and ticked each, then the student would probably get 3 points. If the student found two of the three, s/he'd get 2 points. A student might get 0, 1, 2, or 3 points on this item.

Look now at a snippet from another Data worksheet, would you?:

In this case, language students listened to an audio tape with 10 short sentences, and they also read ten brief paragraphs. The digits in the boxes are now real numbers -- they're not response codes -- the digits represent the number of points the student earned on each of the ten listening items, L1-L10, and the number of points earned on each of ten reading items, R1-R10. The items in this test are said to have been "pre-scored", or "pre-coded".

The following CCs lines were used to process results for the ten Listening items:

*col (c2-c11)

*sub res=(0,1,2,3), name=(Listening AARP test), Title=(L-AARP)

*mws  c2, 0, 1

*mws  c3, 0, 1

*mws  c4, 0, 1, 2

*mws  c5, 0, 1, 2

*mws  c6, 0, 1, 2

*mws  c7, 0, 1, 2, 3

*mws  c8, 0, 1, 2, 3

*mws  c9, 0, 1, 2

*mws c10, 0, 1

*mws c11, 0, 1, 2, 3

The *mws lines indicate that the number of item points possible varied from item to item. For the first two items, scores were limited to 0 or 1; for the items found in columns 4, 5, 6, and 9 possible scores ranged from 0 to 2. Three items, those in columns 7, 8, and 11, had possible scores ranging from 0 to 3.

Fewer lines would have been required had it been possible to get the same number of points on each item:

*col (c2-c11)

*sub res=(0,1,2,3), name=(Listening AARP quiz), Title=(L-AARP)

*mws call, 0, 1, 2, 3

Here, the "call" form of the *mws line has been used -- "call" means "columns all". According to these CCs lines, it's possible for a student to get up to 3 points on each item.

Related tidbit:

Users who often use pre-scored items might want to consider qualifying for use of Lertap's advanced toolbar. Why not get your mouse to click here, and have a read?


Open-ended items

It is possible to have Lertap score open-ended items, such as short-answer questions, and essay questions.

Please refer to the following URL for further information:

http://www.lertap.curtin.edu.au/Documentation/ScoringNonMCItems1.doc


Remove an item.

It sometimes happens that users will be Lertapping along, humming their favourite tune, and suddenly wish that they could omit a single item or two from a subtest, just to see how that might change things (such as the value of coefficient alpha). There are a variety of ways in which this may be accomplished.

The most obvious way is to make a new *col card.

For example, consider these CCs cards:

 *col (c28-c37)

 *sub aff, title=(Comfort)

 *pol +---- ++--+

Now, suppose it was found that the item in column 36 was not correlating well with the others, was serving to bring down the subtest's reliability figure (coefficient alpha), or for some other reason had to be removed from the subtest. The following *col card will do the job:

 *col (c28-c35, c37)

But this isn't the only change required. The *pol card must be changed too -- we've taken an item out of the subtest, and must remove the corresponding plus (+) or minus (-) sign from the *pol card:

 *col (c28-c37)

 *sub aff, title=(Comfort)

 *pol +---- ++-+

An easier way to take the item out is to make use of a special form of the *mws card:

 *col (c28-c37)

 *sub aff, title=(Comfort)

 *pol +---- ++--+

 *mws c36, *

The *mws card above has a single asterisk after the column number. This is a special form of the *mws card, used to remove an item. This special use of the *mws card eases the task of taking items out of a subtest -- there's no requirement to make corresponding changes to other cards, such as the *pol card.

There's another way to remove items from affective subtests: use asterisks on the *alt card, as shown in the example below:

 *col (c28-c37)

 *sub aff, title=(Comfort)

 *pol +---- ++--+

 *alt 55555 555*5

The *alt card above tells Lertap that the penultimate item (second to last) is to be excluded from the subtest.

The examples above are based on an affective subtest, but the special uses of the *mws and *alt cards shown here also apply to cognitive subtests. Consider this example:

 With all items:

 *col (c3-c27)

 *sub res=(A,B,C,D,E,F), Title=(Knwldge)

 *key AECAB BEBBD ADBAB BCCCB BABDC

 *alt 35423 35464 54324 43344 45546

 Using *mws to remove the sixth item:

 *col (c3-c27)

 *sub res=(A,B,C,D,E,F), Title=(Knwldge)

 *key AECAB BEBBD ADBAB BCCCB BABDC

 *alt 35423 35464 54324 43344 45546

 *mws c8, *

 Using *alt to remove the sixth item:

 *col (c3-c27)

 *sub res=(A,B,C,D,E,F), Title=(Knwldge)

 *key AECAB BEBBD ADBAB BCCCB BABDC

 *alt 35423 *5464 54324 43344 45546

SAQ: would the example immediately above actually work? If I copied the 16 lines and pasted them into a CCs worksheet, would they actually work? Yes. This example is just a straightforward job with three subtests. The lines which do not begin with an asterisk are comments, and are not processed by Lertap.

Finally, our examples here have discussed removing a single item from a subtest. To remove more than one item, follow the same pattern. The examples below will remove two items from their respective subtests:

 *col (c28-c37)

 *sub aff, title=(Comfort)

 *pol +---- ++--+

 *mws c29, *

 *mws c36, *

 *col (c3-c27)

 *sub res=(A,B,C,D,E,F), Title=(Knwldge)

 *key AECAB BEBBD ADBAB BCCCB BABDC

 *alt 35423 *5464 54324 4*344 45546

Please note: the special forms of the *alt and *mws cards require Lertap Version 5.2 or better. How to find out your version number? It's simple; a click here will tell you how.

November 2004 note: the new *exc card was added to make it easier to remove, or "exc"lude, items. Please to see the following topic.

Related tidbit:

In June 2004 a special document was created to address a 255-character cell limit in Excel. It contains numerous examples of using CCs lines *alt, *mws, and *wts to remove items from a subtest. If you're connected to the Internet, why not whip out to see this Word file:

ExcelColumnLimitProblem1.doc


Excluding items

Excluding items (*exc)

 Return to Introduction

The matter of removing one or more items from a subtest is discussed in the previous topic. As mentioned there, the quickest way to exclude a single item from a subtest is probably to use a single *mws card. For example, the following CCs line will see that the item resident in column 37 of the Data sheet is eliminated from its subtest:

*mws c37, *

In November 2004 a new CCs card, or line, was added to ease the task of excluding multiple items. Its format is identical to that of the *col card.

The following CCs line will remove, or exclude, the item in column 37:

*exc (c37)

Other examples of the use of this card:

*exc (c12-c14, c42)

Removes the items found in columns 12, 13, 14, and 42.

*exc (c12, c13, c14, c42)

Also removes the items found in columns 12, 13, 14, and 42.

*exc (c17, c21-c25, c27, c35-c40)

Will exclude the items in columns 17, 21, 22, 23, 24, 25, 27, 35, 36, 37, 38, 39, and 40.

Both the *mws and *exc lines may be used to remove items from a subtest, as shown in the example below:

*mws c17, *

*mws c27, *

*exc (c21-c25, c35-c40)

The three lines above will see that the items in columns 17, 21, 22, 23, 24, 25, 27, 35, 36, 37, 38, 39, and 40 are excluded from the subtest.

To see how the *exc card can ease the process of removing items from a subtest, look at the following example:

The example above shows two ways to have 12 items excluded from a 40-item subtest.

The first way involves the use of multiple *mws cards; the second way uses a single *exc card. (There are other ways to exclude items: refer to the previous topic for more details.)


Filtering records

Filtering records (*tst)

 Return to Introduction

The *tst control "card" is a special one, used to have Lertap filter some of the records from whatever data set you happen to be working with. Two examples of the *tst card have been briefly mentioned in the topics above; much more mention of *tst is found in the manual.

In December 2004 the *tst card gained some new smarts. Now the filtering criteria may be more elaborate -- look at these examples:

A) *tst c6=(CS001)

B) *tst c6=(UCS001, CS001)

C) *tst c6=(DL5, DL6, DXL7), c7=(WI)

In example A), all data records with CS001 in column 6 of the Data worksheet will be filtered out into a new data set.

In Example B), all records with either UCS001 or CS001 will be filtered out into a new data set.

In Example C), all records with either DL5, DL6, or DXL7 in column 6, and WI in column 7, will be filtered out into a new data set.

So, what are the "new smarts" we're talking about? Well, it used to be the case that the filtering criteria could consist of just a single character, as in, for example, *tst c2=(W) and *tst c4=(B,G). Now the criteria may be any length. (There may be up to 20 criteria for each column. In the examples above, in B) there are said to be two filtering criteria on column 6, while in example C) we'd say that there are three criteria on column 6.)

In June 2006 the Move+ Menu was blessed with a new option, Recode a Data column, which provides another way to filter records. Now the word "Delete" can be attached to certain data records, as can the word "Exclude". In some cases this may be a very effective way to quickly remove records. Click here.


How CCs cards work

Each subtest requires a minimum of two CCs cards.

Cognitive subtests

Cognitive subtests must have a *col card, and they must also have a *key card with the right answers.

The number of columns mentioned on the *col card tell Lertap the number of items in the subtest. The default response code set for cognitive items is Res=(A,B,C,D).

The CCs card order for cognitive subtests is:

*col  (a required card)

*sub

*key  (a required card)

*alt

*wts

*mws

The *key, *alt, and *wts cards require a character for each item in the subtest. If there are five items, there must be five entries on the *key card; the *alt and *wts cards, if used, must also contain five entries. There may be spaces between the entries. We like to put a space after every five entries as that way the card is easier to read; we also like to use a fixed-pitch font with CCs cards, such as Courier New, so that when we use *key, *alt, and *wts cards the entries on all cards line up.

*mws cards are unique in that they correspond to just a single item.

Affective subtests

Affective subtests must have a *col card, and they must also have a *sub card with the AFF control word on it.

The number of columns mentioned on the *col card tell Lertap the number of items in the subtest. The default response code set for affective items is Res=(1,2,3,4,5).

The CCs card order for affective subtests is:

*col  (a required card)

*sub  (a required card; must have AFF on it)

*pol

*alt

*mws

If used, the *pol and *alt cards require a character for each item in the subtest. If there are five items, for example, there must be five entries on these cards (if they're used).

*mws cards are unique in that they correspond to just a single item.

Control words

The *sub card may have a number of control words on it; these are summarised in the tables below:

*sub control words for cognitive subtests:

CFC | Optional. Means "correction for chance". Adjusts subtest scores for the (estimated) effects of guessing.
Mastery | Optional. Gets Lertap to produce its mastery test analysis and report. Also acts as if the PER control word had been used, causing the PER score to appear on the Scores report. (Note: this control word is the same as using Mastery=70; a default mastery level of 70 is programmed into the System worksheet.)
Mastery= | Optional. Lertap will produce a mastery test analysis and report, using the cutoff figure found after the equals sign. Also causes a PER score to be created. Example: Mastery=65
MDO | Optional. Gets Lertap to turn off its missing data item scoring. Causes a non-response to be scored as zero points. (A click here will whisk you away to more about MDO.)
Name=() | Optional. Whatever is found between the parentheses is used as a header on some of Lertap's reports, such as Stats1f and Stats1b. While the header can have any length, something less than 40 characters is best. Example: Name=(SOC 505 FINAL, November 2003)
PER | Optional. Causes a percentage of maximum possible score to be created; this will appear as a column in the Scores report.
Res=() | Required if the default Res=(A,B,C,D) setting is not appropriate.
SCALE | Optional. Causes a z-score to be computed and added as a column in the Scores report.
Title=() | Optional. Provides a short label for the subtest score. This will appear as a header at the top of a Scores column. Should be no longer than 8 characters. Example: Name=(SocFinal)
Wt= | Optional. Applies only when there are multiple subtests. Determines how the subtest's score comes into the total test score. Example: Wt=0.5

*sub control words for affective subtests:

AFF | Required. The appearance of the AFF control word on a *sub card is the only way Lertap knows a subtest is of the affective type.
MDO | Optional. Gets Lertap to turn off its missing data item scoring. Causes a non-response to be scored as zero points. (A click here will whisk you away to more about MDO.)
Name=() | Optional. Whatever is found between the parentheses is used as a header on some of Lertap's reports, such as Stats1f and Stats1b. While the header can have any length, something less than 40 characters is best. Example: Name=(WA/Maui beaches survey, January 2004)
PER | Optional. Causes a percentage of maximum possible score to be created; this will appear as a column in the Scores report.
Res=() | Required if the default Res=(1,2,3,4,5) setting is not appropriate.
SCALE | Optional. Causes a new score to be computed and added as a column in the Scores report. The new score is the original score divided by the number of items in the subtest.
Title=() | Optional. Provides a short label for the subtest score. This will appear as a header at the top of a Scores column. Should be no longer than 8 characters. Example: Name=(BeachSur)
Wt= | Optional. Applies only when there are multiple subtests. Determines how the subtest's score comes into the total test score. Example: Wt=0.75

The order of the control words is not important. For example, the following two cards accomplish the same thing:

*sub Title=(NewsQuiz), PER, SCALE

*sub SCALE, Title=(NewsQuiz), PER

The control words may be abbreviated, or expanded. The following cards result in the same actions:

*sub AFF, Name=(Beach survey 1), Title=(Beachin)

*sub Affective, T=(Beachin), N=(Beach survey 1)


Toolbar

There are different versions of Lertap 5, each with its own main toolbar.

The toolbar shown below is the standard one seen on Windows computers, as of April 2006:

This toolbar has 9 icons, and 7 drop-down menus. The icons result in a single action, while the drop-down menus, when clicked on, produce a list of options, each of which may be clicked on.

The toolbar image above is "hot". Move your mouse over it, and click to see what the toolbar's icons and drop-down menus do.

The equivalent toolbar, when seen on a Macintosh computer running OS 10.4.3 and Excel X, looked like this as of June 2006:

How to get the toolbar to display on your computer's screen? Find and open the file named Lertap5.xls.

That's all? Yes, and no. Some users find that opening the Lertap5.xls file does not cause the toolbar to show on their screen. This happens when Excel's macro security level has been set to high. The fix is easy -- use Excel's Tools menu, click on Macro, click on Security, and select Medium or Low. Then close the Lertap5.xls file, and re-open it. (Click here for related comments.)

Some users ask why the Lertap toolbar does not appear whenever they open a Lertap workbook. The reason is that the toolbar is only associated with the Lertap5.xls workbook. Creating a Lertap workbook does not result in a workbook which has the toolbar attached to it.

To use the toolbar with a Lertap workbook, open the Lertap workbook, and then also open the Lertap5.xls file. After following these steps, some users say that they can no longer see their Lertap workbook -- they only see the Lertap5.xls workbook. In this case, use Excel's standard toolbar, and click on Window. A list of all open workbooks will appear.


New Menu

The New menu is the best way to create a new Excel workbook ready to work with Lertap. It's not the only way, but it's the best. Why? Because it results in an Excel workbook set up to display Lertap's preferred fonts, and a workbook with the two core Lertap worksheets, Data and CCs.

To find out what the New menu's options do, click on the topics shown in the box below:

It is possible to create a Lertap workbook without using the New menu options. All that's required by Lertap is an Excel workbook with data records in a worksheet named Data, and control lines (or "cards") in a worksheet named CCs.

If you presently have an Excel workbook with data records in it, you can try to rename the worksheet with the data records to Data, and insert a new worksheet with a name of CCs. That will work. However, font problems may develop when Lertap's Run menu options are taken -- Lertap's reports, such as that seen in the Stats1f worksheet, may turn out to be poorly formatted.

If this happens, a suggestion is to use the New menu to "Make a new blank Lertap 5 workbook". Then, from the original workbook, copy all data records to the blank workbook's Data worksheet.

Click here for a definition of a Lertap workbook.

The New Menu is discussed in Chapter 3 of the manual, under the section titled "Setting up a new workbook".

In the printed manual, the reference is page 60.


Blank workbook

Make a new blank Lertap 5 workbook

This option opens a new Excel workbook with two blank worksheets. One of the worksheets is named Data, while the other is named CCs.

The Data worksheet is split after Row 2 (the worksheet is fixed so that the top two rows always display header information).

Note that the new workbook should be saved as soon as possible. Excel's File menu options are used for this.

Click here for a definition of a Lertap workbook.


Present Data headers

New workbook with present data headers rows

This option creates a new Excel workbook with Data and CCs worksheets which are empty of data records, but otherwise identical to the original workbook.

The Data worksheet's first two rows, the header rows, will have information copied from the original workbook's Data worksheet.

The lines in the new workbook's CCs worksheet will be identical to those in the original workbook's CCs worksheet.

Note that the new workbook should be saved as soon as possible. Excel's File menu options are used for this.

Click here for a definition of a Lertap workbook.


Copy present

New workbook which is a copy of the present one

This option creates a new Excel workbook which has Data and CCs worksheets copied from the original workbook. If the original workbook has more worksheets, they are not copied by this option -- only the Data and the CCs sheets get copied.

Given that this option only copies Data and CCs worksheets, how is a complete copy of a Lertap workbook made? With the original worksheet open, a complete copy may be made by using the Save As... option, found under Excel's File menu. Or, when the workbook's name is seen in a folder listing, right-click on the name, and then select Copy.


Data Entry Aids

There are two aids to make data entry easier. They're represented by two icons on Lertap's toolbar: DataEntryAids.

Both of these icons will take a string of item responses, and then dissect the responses, distributing them one by one over the cells to the right.

DataEntryAid1 this data entry aid operates by opening up an input box for the string of responses to be typed into, as shown below:

The string of 25 letters and 10 digits seen in the input box (starting with BECDD ...) will be taken apart, and distributed over the cells to the right.

To the right of where? To the right of the active Data worksheet cell; usually Excel highlights the active worksheet cell by putting a box around it.

DataEntryAid2 this icon, when clicked on, activates "The Spreader", a powerful data entry aid. To understand how it works, look at the screen capture below:

A string of 35 characters has been entered into a single cell, directly on the Data worksheet. When you employ The Spreader, it will almost always be the case that row after row will have its item responses entered in this manner. After the last row of data has been entered, you scroll up to the first row, click on the cell containing the string, and then click on The Spreader's icon, DataEntryAid2.

The Spreader will dissect the string, spreading the characters, one by one, over the cells to the right. Having done this, it then looks at the next row in the worksheet. If there's another long string to be dissected, it does so. The Spreader continues to work down the rows, stopping when it finds one without a string.

The Spreader may be stopped at any time by pressing the Esc key on the computer's keyboard.

What about missing data? Say someone has not answered one of the items -- in this case, leave a blank in the string (pressing the space bar on the computer keyboard will produce a blank).

An important case arises when the string of responses begins with a digit. Excel will think the entry is going to be a number, and strange things can happen. When the string begins with a digit, it should be preceded by an apostrophe, as seen here:

'2344133124AADDB

The apostrophe tells Excel to consider the entry as "text", not a number. Another way to define an entry as text is to use Excel's Format menu / Cells, then select Text. The cells in an entire column may be formatted as text by selecting the column before using the Format menu.

The Spreader is a favourite tool at Lertap HQ. It's powerful indeed. It is often useful when importing data from other applications, including scanners.

The data entry aids are discussed in Chapter 3 of the manual, under the section titled "Entering item responses".

In the printed manual, the reference is page 61.

Update note:

In September 2003 The Spreader gained more smarts. Its standard method of operation now involves two passes down the cells whose contents are to be spread.

In the first pass, the length of the cell's string is compared to the length of the string in the cell above. If the lengths are not the same, The Spreader sounds an alarm. You get the chance to stop The Spreader, or to continue. If you choose to stop, you can then edit the cell.

Once The Spreader has worked its way down all the relevant cells the first time, it will then ask if you're ready to truly spread cell contents. You can stop at this point without anything having changed. If you elect to continue, The Spreader goes back up to where you last started it from, and spreads things to the right

You can change the way The Spreader operates by changing the appropriate row in the System Worksheet. It's possible to tell The Spreader to forget about the first pass, the one where it checks string lengths. The Spreader is not slow, but, as you'd expect, it runs even faster if it doesn't have to make two passes.

At Lertap HQ we favour the two-pass method of operation as we feel it's reassuring to check string lengths. However, even the two-pass Spreader will not control for a nasty problem: when the string has been created by importing data from a scanner-made text file, blanks at the start of the string may be lost. Such blanks usually correspond to unanswered questions. It's rare for a respondent to leave the initial questions unanswered, but it does happen, and when it does real care is required to make sure that the blanks remain at the start of the string. If they do, The Spreader will spread them. If they don't, woe! -- the string will be shifted to the left, and Lertap will be unaware of what's happened. If a test is being scored, the score will be wrong.

There's a bit more about this nasty in the Import & Export topic.


Run Menu

The Run menu provides paths to Lertap's data analysis routines. It's used after all data have been recorded in the Data worksheet, and after control lines, or "cards", have been typed into the CCs worksheet.

A typical data analysis procedure consists of two steps. First, users click on the "Interpret CCs lines" option. This gets Lertap to check the CCs lines for syntax errors. If no errors are found, Lertap goes on to produce the "Freqs" report, that is, a worksheet which summarises the response frequencies found in the columns of the Data worksheet. Which columns? The ones referenced by the *col lines in the CCs worksheet.

As part of the "Interpret CCs lines" process, Lertap also writes some intermediate worksheets with data required by the Elmillon program. These are the "Sub" worksheets -- users with a keen eye can see the Sub worksheets being formed as Lertap goes about its business, but then, just before focus shifts to the Freqs worksheet, the Sub worksheets are hidden from view. Users usually have little need to see them; however, they're not secret -- they can be unhidden. There will be one Sub worksheet for each of the CCs worksheet's *col lines.

The second of Lertap's usual two-step data analysis process involves clicking on the "Elmillon item analysis" option. This is the option which creates the various statistical reports which are Lertap's reason for being. Each of these reports is an Excel worksheet; they have names such as Stats1f and Stats1b.

Elmillon also produces scores, one for each subtest, or scale.

For more information on what the Run menu's options do, click on the topics shown in the box below:


Interpret CCs lines

Before this option is selected, it must be the case that (1) data records have been created in the Data worksheet, and (2) control lines, or "cards", have been placed in the CCs worksheet.

The first time this option is clicked on, Lertap gets Excel to read the responses found in the Data worksheet, looking in the columns specified in the *col lines of the CCs worksheet.

If there are no errors in the CCs worksheet, Lertap and Excel will produce new worksheets. One of them is called "Freqs", for frequencies.

Usually, the "Elmillon item analysis" option will be taken next. This results in even more new worksheets, with names such as Scores, Stats1f, Stats1b, and so on.

Note added May 2005: it is now possible to get Lertap to automatically roll through from "Interpret CCs lines" to "Elmillon item analysis" without stopping, as detailed in the following topic: Production mode.

If the "Interpret CCs lines" is later selected again, a warning message will appear, saying that there's a possibility of losing data. This is purely precautionary -- Lertap is about to delete Freqs, Scores, Stats1f, and so forth -- but this is usually only natural; new versions of these same worksheets will be generated, which is almost always what users want. (Lertap is being overly cautious in sounding this warning; the primary worksheets, Data and CCs, are never affected by this action.)

For a related topic, see "deleting secondary worksheets".

Page 142 (Chapter 9) of the printed manual also discusses these matters.


Production mode

A "production mode" capability was added in May, 2005.

Have a look at the following snapshot of lines 35 through 38 of the System Worksheet:

If the production mode setting is set to "yes", then Lertap will not stop after it has been requested to "Interpret CCs lines" -- it'll power ahead, automatically activating the next option on the Run menu, "Elmillon item analysis".

There are three other "yes" settings which may be made in these rows -- you see them above. You can get Lertap to automatically output histograms for each subtest, response charts (bar charts for affective subtests, quintile plots for cognitive subtests), and an item scores matrix (IStats). This gives you time to sit back with your cup of coffee, and watch the screen flash before you as Lertap goes about its tasks.

Of course, this is not recommended. Lertap ships with all the production mode options set to "no". Why? Because sound data processing practice is always supposed to involve a data integrity check or two. You want to make sure the data you're feeding into Lertap has been subjected to some quality control. In Lertap 5, the main means of doing this is by having a careful look at the Freqs report, the worksheet produced by taking the "Interpret CCs lines" option from the Run menu.

The Freqs report readily indicates the characters found in each of the data columns. If you're running with a cognitive test whose options use the letters A, B, C, and D, then you'll want to check down the Freqs report to make sure that no other characters have crept into the scene, such as, perhaps, lower-case letters (a, b, c, or d).

If your scan of Freqs reveals strange characters, your task is to find them, and to fix 'em. It's only after you've done this that you'll go on to get results by activating Elmillon.

But there are times when users do not concern themselves so much with Freqs. For example, many users have their data prepared by using a scanner. Scanners can readily be trained to hoot and holler when they encounter bad data. Oft times scanners will output a special character, such as an asterisk, to signal a questionable result.

For more on the System worksheet, click here. The production mode options are all dynamic ones -- they take effect immediately, without requiring that Lertap first be closed and then reopened.

Note that the yes / no entry in row 26 controls the following three lines. If row 26 is set to "no", then that's it -- the following three rows are ignored, even if they have "yes" entries.

Finally, there will be users who'd like to have two versions of Lertap: one as shipped from our manufacturing plant, with production mode off, and another version with production mode on.

Can do, emu? Yes. A single computer, Windows or Macintosh, can have multiple copies of the Lertap5.xls file. Of course, each copy has to be in a distinct folder (subdirectory), but that's easy. If you're going to do this, and you're running under Windows, then you might like to copy the LRTP5HHelp.chm to each folder too -- it's the Lertap help file, "Lelp". If you have Lertap5.xls in a folder, without its corresponding Lelp file, then Lertap help will not be available whenever you try to access it from within Lertap (this comment applies only to Windows users).

Related tidbit:

For more about this topic, get productive: see "ScannerEjemplo1.doc", a marvelous, captivating Word document with lots of hints for what to do on rainy days, available via the Internet: click here if you've got connections.


Elmillon

Elmillon item analysis

Elmillon is the name of Lertap's main item analysis routine, first developed for the Venezuelan Ministry of Education. The name has origins in the Spanish language: "un millon", or, in English, "thanks a million", is what the chief of the data analysis section, Rogelio Blanco, said when the routine was debugged and delivered. The name Elmillon has been used since then.

Elmillon is always run after the "Interpret CCs lines" option has been taken. Elmillon reads data, makes subtest and scale scores (written to the Scores worksheet), and goes on to produce from one to three item analysis reports. The item analysis reports are Excel worksheets, having names like Stats1f, Stats1b, and Stats1ul.

Once Elmillon has been run, it will not run again until the workbook's secondary worksheets have been deleted. This isn't as difficult as it may sound: selecting "Interpret CCs lines" will delete them.


External criterion

External criterion analysis

The item correlation coefficients which form part of Lertap's Stats1f and Stats1b reports are based on correlating item scores with what's referred to as an "internal criterion": the number produced by scoring the remaining items in the subtest or scale to which the item belongs.

It is possible to replace the internal criterion with another score. This other score is referred to as an "external criterion". An external criterion may be used as part of the process of validating test items.

In Lertap, the external criterion score must correspond to a column in the Scores worksheet. The analysis begins by having the user pick the Scores column that has the score to be used as the external criterion. Once this is done, Lertap asks the user to pick out the subtest which has the items to be correlated with this score. This is done by displaying Sub worksheets.

Users of the external criterion analysis feature will often have a criterion measure which needs to be imported to the Scores worksheet. In this case, the criterion measure should be recorded in a column in the Data worksheet. Once it's there, the Move menu on the Lertap toolbar will allow the measure to be copied over to the Scores worksheet.

The effects of part-whole inflation may be examined by using an external criterion analysis. Lertap's item correlation coefficients are always corrected for part-whole inflation (sometime referred to as part-whole contamination); to see what they'd be without such correction, define a subtest's score, as found in the Scores worksheet, to be the external criterion.

Related tidbits:

See the "Using an external criterion" section of the manual's Chapter 8 (p. 134 in the printed manual).

Also see "Using Lertap in a Test Validity Study", a 12-page Word document available via the Internet: click here if you're connected.


Output item scores

Output item scores matrix

A person gets a score for each item s/he answers. For example, for a cognitive item, the custom is to give one point for a right answer, and zero points otherwise. For affective items, each possible response usually has a certain number of points associated with it. "Strongly agree", for example, might equate to a score of 5 points, while "strongly disagree" might be made equal to just a single point.

In Lertap, a person gets a score on each item even when no answer is given. For cognitive items, a non-response usually equates to zero points, while for affective items Lertap will apply MDO, the "missing data option", to non-responses. MDO usually equals the mean of the possible scores for an item -- for example, if the possible range is 1.00, 2.00, 3.00, 4.00, 5.00, then MDO=3.00 (note).

When asked to do so, Lertap will get all the item scores together, and write them to a new worksheet named IStats. An IStats worksheet has three sections. The first section has rows whose cells contain the score each person earned on each item.

The second section includes rows of descriptive statistics for each item, such as the median, mean, and standard deviation.

The third section involves the creation of inter-item correlations. Under "normal" conditions, a single matrix of Pearson product-moment correlation coefficients is made, with a row of average correlations included at the bottom of the matrix. Lertap uses standard Excel functions for all of these calculations. Average correlations are computed by using the n-1 non-diagonal entries in each column, where "n" is the number of items.

There are two settings in the System worksheet which affect the information found under this third section of the IStats worksheet. One of them replaces the diagonal element of the correlation matrix with the "SMC", the squared multiple correlation. The SMC for an item is often used in factor analysis as an initial estimate of the item's common variance.

Prior to October 2004, Lertap used Excel's MINVERSE matrix function to invert the correlation matrix, part of the process of calculating SMC values. It was found, however, that MINVERSE would regularly fail when asked to invert matrices with 50 or more subtest items, and a switch was made to the M_INV function found in the Foxes Group's package of matrix routines. (For more about the Foxes Group, see the eigenvalue topic.)

Both MINVERSE and M_INV require a scratch area to work with, and Lertap uses the IStats worksheet itself for this (on slow computers, or cases where many items are involved, you might see Lertap scratching about).

The standard setting for SMC calculations is "off"; to activate SMC output, change the setting seen in Row 21, Column 2 of the System worksheet. If this setting is changed, it takes effect immediately. (However, this doesn't mean that any correlations matrices you may have already made will immediately change; in fact, they won't -- you'll have to delete or rename the IStats worksheet, and then get Lertap to make a new IStats sheet.)

For additional information about Lertap's calculation of SMC values, just page ahead to the next topic.

Other settings in the System worksheet determine whether or not Lertap might add a matrix of tetrachoric correlations to the IStats output, and possibly make worksheets designed for export to other data analysis systems. There's more about these matters in the following topics.

It's also possible to get your favourite item analysis package (Lertap, natch) to make a stab at determing the eigenvalues of the correlation matrices it produces. This is also discussed in an ensuing topic.

For more about IStats sheets, please refer to the manual. You'll find an example in Chapter 7, and a fairly thorough discussion under the "Item scores matrix" section of Chapter 10. Note, however, that the SMC, tetrachoric, export worksheets, and eigens options were added after the manual was printed.


SMCs

An item's SMC value, its squared multiple correlation, indicates the proportion of the item's variance which may be linked to, or predicted from, the other items in the subtest. As mentioned in the previous topic, the SMC is sometimes used as an estimate of the amount of variance any single item has in common with the other items.

For technical discussions on the SMC, and its calculation, see Pedhazur & Schmelkin (1991, pp. 414-417), Lord & Novick (1968, pp. 265-266), Hays (1973, pp. 705-708), and Glass & Stanley (1970, pp. 186-191), or search the Internet for "multiple correlation coefficient". (To get to the Lertap references page, begin with a click here.)

In October 2004, Lertap's IStats worksheet was modified so as to give some prominence to SMC values. Help your little self to a squiz of the following sample:

As seen above, the IStats report now has a row with SMC values (just above the eigens row), followed by a display of SMC bands.

The bands give a quick idea of the spread of SMC values -- we see, for example, that nine subtest items had an SMC value equal to or greater than 0.60, but less than 0.70. Among these nine were Q1, Q2, Q3, Q6, and Q8 -- you can see the actual SMC values for these five items by looking at the row of SMC values showing above the bands.

In this case, we see that Q2's SMC was 0.67. We may interpret this as meaning that 67% of Q2's variance can be explained by the other items in the subtest. If we took the square root of Q2's SMC, we'd get 0.82, a value we may interpret as indicating the correlation between Q2 and the other items. (In somewhat more technical lingo, 0.82 is the value of the Pearson product-moment correlation coefficient between the scores people earned on Q2 and a specially-weighted linear composite score formed from the other items. The "special weights" are determined via a multiple linear regression analysis, as the references point out.)

Now, you'll remember that there is another Lertap report which indicates how an item correlates with the other items in the subtest.

There is?

Sure. Look at the item discrimination bands seen in the following screen snippet:

In Lertap, the standard index of item discrimination (or, for affective subtests, the item correlation) is the correlation between the item and a person's score derived by summing over all the other items in the subtest. (There is more about this in the manual.)

Note where Q2's discrimination index falls: in the 0.60 band. If we could scroll up the Stats1f report, or page over to the corresponding Stats1b report, we'd find Q2's correlation to be 0.66.

We have, then, two measures of Q2's correlation with the other subtest items, 0.82 and 0.66. Both figures represent the correlation between Q2 and a composite score formed by adding together the scores on the other subtest items; the SMC-based correlation will always be equal to or greater than the other correlation as it, the SMC-based value, is derived by using the special weights resulting from the multiple linear regression analysis underpinning the calculation of the SMC.

What do you have to do to get Lertap to produce SMC values? Nothing much, really -- as of October 2004, whenever you use the Run menu's option to "Output item scores matrix", the resultant IStats report will automatically include the row of SMC values, and the little table with SMC bands, as seen above.

What, then, is the SMC setting mentioned in the previous topic? It's a setting which determines whether or not the diagonal values of the IStats correlation matrix has 1's or SMCs. Why do some users want to have SMCs on the diagonal? Often because they're thinking of using the correlation matrix as input to a factor analysis program.


Tetrachoric correlations

Tetrachoric correlation coefficients are computed when two conditions are met: (1) the tetrachorics option is set as "Yes" in the System worksheet, and (2), Lertap finds that the item scores are just zeros and ones.

These conditions are in fact easy to satisfy. The tetrachoric option's default setting in the System worksheet is No when Lertap is first installed, but this may quickly be changed to Yes. And cognitive test items are very often scored on just a right/wrong basis, with one point for a correct answer, zero points otherwise.

What are tetrachoric correlation coeffcients? They're estimates of what the correlation between two items would be if responses to the items had an underlying normal distribution, instead of the simple right/wrong dichotomy used to score the items. Some researchers and test developers are at times willing to assume underlying normal distributions, especially when they are interested in aspects of IRT modelling.

For more reading, use Lertap's references page, looking at Crocker and Algina, Lord, and/or Glass and Stanley. Or, search the Internet for definitions and discussions.

To compute the tetrachorics, Lertap uses an algorithm created by Brown (1977) (see References). Brown's algorithm calls for the use of two normal-curve functions: "AlNorm", and "PPND". Lertap uses two in-built Excel functions instead: NORMINV and NORMSDIST.


A DAT-like worksheet.

The Bilog and Bilog-MG computer programs like to have input formatted as what their authors call a "DAT" file, by which is meant a simple unformatted text file with fields of fixed length, suitable for reading by a FORTRAN Input statement. (Such files are often referred to as 'ASCII' files.)

Bilog-MG assumes that items have been scored on a dichotomous basis, that is, as either right or wrong, with a "1" used to signify right, and a "0" (zero) used for wrong.

A Bilog-friendly DAT worksheet will be created by Lertap when the appropriate option is set in row 23 of the System worksheet.

Here's a screen shot from a Lertap DAT worksheet, one from a data set having 25 cognitive items, using a record ID field with a number between 15001 and 28000:

Every time Lertap makes a DAT worksheet, it inserts a Fortran format statement at the top. The line above says the data records start with a 5-column ID field, followed by a space, followed by 25 item scores.

Here's a snippet from another DAT worksheet. This one corresponds to a data set with 45 cognitive items, and an ID field with student names. The longest student name was 14 characters wide; Lertap has right-justified the names, using blanks on the left side whenever the name was shorter than 14 characters:

Now, a program such as Bilog, Bilog-MG, or XCALIBRE will not read data from an Excel worksheet. The sheet has to be saved as a text file. How? How to save the DAT worksheet as a text file? Click here to find out, or page ahead to the topic titled "Creating a text file".

Once the DAT worksheet has been saved as a text file, the first line, the Fortran format statement, should be deleted -- Bilog won't like it. However, as long as the original DAT worksheet remains part of a Lertap/Excel workbook, the Fortran format line should not be deleted -- it's possible to add more data to the DAT worksheet, and the procedure which does this will cough, sputter, and die if the format line has gone walkabout.

What's this about adding more stuff to the DAT worksheet? Use the "Copy a Data column to the DAT worksheet" option (of course!), as found under the Move+ menu.


An XCALIBRE worksheet

XCALIBRE is the name of an IRT program created by Assessment Systems Corporation, ASC, in the United States.

Lertap has an XCALIBRE interface which works much like that for the Bilog-MG program mentioned in the previous topic. To activate the interface, go to the System worksheet in the Lertap5.xls file, and activate the setting in Row 24, Column 2 -- set it to "yes". Once "yes" is in the appropriate spot in the System sheet, Lertap will spin out a new worksheet every time the "Output item scores matrix" is selected from the Run menu, providing that the subtest being processed is a cognitive one. The worksheet will be named "XCal".

Like Bilog-MG, XCALIBRE wants to have its input arranged in a very specific format. Here's a screen shot of a Lertap XCal worksheet, one resulting from processing a 25-item cognitive subtest

The first four rows of the XCal worksheet have the control information wanted by the XCALIBRE program. The 2nd row has the keyed-correct answers for the items; the 3rd indicates how many options were used by each item; and the fourth, a row of Ys, tells XCALIBRE that all items are to be included in its analysis.

What about the 1st row? It's really the most detailed, having four fields of critical information for XCALIBRE. The first field, characters 1-3 in the row, give the number of subtest items. The second field must appear in character position 5; this field tells XCALIBRE the code used in the data records to indicate an omitted item -- Lertap follows the XCALIBRE convention of using the letter O for this code, but you may change it as wished. The N following the space after the O has to appear in character position 7; it indicates the code for items which have not been reached by a respondent. The final control field appears in character positions 9-10, giving the maximum number of characters of ID information for each test taker. In the example used here, only two ID characters were used.

The actual data records begin in Row 5 of the XCal sheet. Each record has its ID code as the first characters, followed by the item responses. Note that the screen shot above has blanks in some of the records -- a dinkum XCALIBRE user would replace these with the code used to indicate omitted items -- in this example that would be the letter O.

XCALIBRE users might want to note the caution given in the XCALIBRE manual about processing "extremely large data sets", and avail themselves of Lertap's all-conquering "To Halve and Hold" option, an option which randomly splits a data set into halves.

Once Lertap has made its XCal worksheet, are you all set to run the XCALIBRE program? No. The XCALIBRE program will not read an Excel spreadsheet. The XCal worksheet has to be saved as a text file, and XCALIBRE wants the text file to have an extension of "DAT". You now have a need to know how to save an Excel worksheet as a text file, and we've got some comments ready for your peepers to peep -- click here to jump to them, or simply page ahead until you get to the "Creating a text file" topic.


RSAdata worksheet

An RSAdata worksheet

RSA stands for "response similarity analysis" -- you'd be interested in RSA if you wanted to see if the responses of any two test takers were, as Wesolowsky (2000) would say, "excessively similar". In less diplomatic terms, RSA is used, by some, to examine the possible presence of cheating in an examination environment.

Lertap will produce a worksheet, "RSAdata1", and a special text file, "SCheckData1.DAT", whenever users take the "Output items scores matrix" option from the Run menu, and have set the RSA option to "yes" in the System worksheet. If production mode is on, and your data set includes more than one subtest, then there will be additional files: RSAdata2, SCheckData2.DAT, and so on (one pair of files for each subtest).

The screen snapshot below captures the System worksheet's RSA settings as found in July, 2005 -- note the "yes" setting in row 25.

The snapshot below indicates how the RSAdata worksheet will generally look:

The RSAdata worksheet is made to conform to a format used by Wesolowsky's "SCheck" program. (Refer to the references for the appropriate citation to Wesolowsky's work in this area, and to his web page for information about SCheck software: http://www.business.mcmaster.ca/msis/profs/wesolo/wesolo.htm.)

Each row in the RSAdata worksheet contains seven columns of information.

The first column corresponds to the Lertap ID in use -- Wesolowsky generally refers to this as the student ID number, but it doesn't have to be a true number -- it can be a name.

The second column is referred to as "name" in SCheck; Lertap inserts "DataRowX" instead, where X corresponds to the row number in the Data worksheet.

The third "can be initials", according to the SCheck.exe user guide -- Lertap leaves this column empty.

The fourth column's contents has a length equal to the number of items in the subtest, and indicates how each student responded to the items: a full stop (or "period") indicates that the student got the item right; a dash (or hyphen) indicates that the student did not answer the item (or had an answer not recognised by Lertap); and a letter or a digit indicates which wrong answer, which distractor, was selected.

In the snapshot seen above, the first student has an ID of 9; his or her complete data may be found in row 3 of the Data worksheet. The student left four items unanswered (there are four dashes), and got only three items correct (there are three full stops). The student selected distractor C on the first and second items, D on the third, B on the fourth, and so on.

Columns 5, 6, and 7 indicate the number of items a student answered correctly, the number answered incorrectly, and the number of questions having what Lertap refers to as an "other" response -- this is usually the same as the number of unanswered questions.

The red triangles (which may appear black on your screen or printout) seen in the snapshot indicate that the worksheet cells have comments. If you were to hover your mouse over one of the cells, you'd see that the comment is "Pickable for similarity analysis". RSAdata cells which have a red triangle will be included whenever the "RSA similarity analysis" option is taken from the Run menu. (This option has nothing to do with the SCheck program.)

Besides the RSAdata worksheet, Lertap will produce a companion "ASCII" file called SCheckData.DAT. If you've saved your workbook prior to selecting the "Output items scores matrix" option from the Run menu, Lertap will produce a message such as the following:

The purpose of this message is to remind you that you've now got a brand new DAT file to work with, and to tell you where to find it. If you haven't saved your workbook prior to selecting the "Output items scores matrix" option from the Run menu, this message may be a bit ambiguous, but the new ScheckData.DAT file will be on your computer, somewhere. (Please note that this message will not appear if you have Lertap set to run in "Production mode".)

Lertap's SCheckData.DAT file is ready for input to Wesolowsky's SCheck software. The contents of this file are very similar to those of the RSAdata worksheet, with commas used to separate information:

9,DataRow3, ,CCDBA.ACA.C-A-D.-BDA-EEBF

31,DataRow4, ,BA..A...E....BDA..AD.BDBE

26,DataRow5, ,C.D...A.F.D.....AA.DC-.CE

27,DataRow6, ,..A..CA.-AC.....AA.CABDBB

21,DataRow7, ,...B.CA.AA.-...A...DAB..E

(The information from columns 5, 6, and 7 of the RSAdata worksheet is not carried over to the SCheckData.DAT records.)

An option on Lertap's Run menu, "Response similarity analysis (RSA)" will get Lertap to use the RSAdata worksheet to feed its own response similarity investigator. To read more about this, simply click here.

To read more about the System worksheet, give a wee click here.


Eigenvalues

The eigenvalues, or "latent roots", or "characteristic roots", of a correlation matrix are sometimes used as a means of estimating the number of factors (or components) which may underpin a test, or a scale. There are often times when researchers would like to be able to say that their test is unidimensional, involving a single factor or construct. Some feel that a test may be said to be unidimensional if it can be shown that the largest eigenvalue underlying the test's correlation matrix is so dominant that it dwarfs the others. (See references and discussion below.)

Eigenvalues are computed if the System worksheet has "yes" in Row 22, Column 2.

Lertap's eigenvalue extraction uses computational routines produced by Leonardo Volpi and the Foxes Group in Italy, made available by the authors' kind permission. The Foxes Group's general matrix package, "Matrix.xla", is freely available at: http://digilander.libero.it/foxes/index.htm. Matrix.xla is a powerful, extensive set of matrix manipulation routines for use with Excel; it includes the ability to produce a complete principal factors / components analysis, with Varimax rotation, something Lertap users may wish to experiment with.

Here's a sample of Lertap's output with "eigens":

In this example, the 10-item "Comfort" affective scale seen in the Lertap Quiz data set, the largest eigenvalue was 3.83, the smallest 0.16. In a well-conditioned correlation matrix with 1's (ones) on the diagonal, the sum of the eigenvalues will equal n, the number of test items (assuming the correlations are Pearson product-moments, not tetrachorics).

The row with the actual eigenvalues is followed by the "percent" row seen above. The percent figures appear whenever the correlation matrix has 1's on its diagonal; when the SMC setting is on, and SMCs are found on the diagonal, two changes are made to the table: the percent figures are not created, and the correlations found in the p-comp1 row are replaced with correlations between the item and the first principal factor, with the row's label then changing to p-fact1.

What do the percent values mean? Well, first note that there are ten items in this example, Q26 through Q35. There are also ten eigenvalues. As noted above, the sum of the eigenvalues equals the number of items: 10 in this example. The percent value for the first eigenvalue is 100(3.83/10), or 38.3%.

Each eigenvalue corresponds to what's called a "principal component". If we could look at the multivariate scatterplot of the ten items, and if each item had a distribution meeting the requirements of the normal distribution, the scatterplot would have the form of an n-dimensional ellipsoid, where n is the number of items (10 in this case). If the items are uncorrelated, the ellipsoid is an n-dimensional sphere. If, on the other hand, the items are correlated, the sphere stretches out to an ellipsoid.

After the percent row comes the "p-comp1" row, giving the correlation of each of the items with the first principal component -- the values found in this row are also sometimes called the "loadings" of the items on the first principal component.

The first principal component corresponds to the ellipsoid's major axis, to its longest axis. Each eigenvalue represents the relative length of one of the ellipsoid's axes. Each of these axes is said to represent, or correspond to, a principal component.

Think for a moment of the case when n=3. If the three items are normally distributed and uncorrelated, their scatterplot will have the form of a soccer ball, a perfect sphere. As the three items begin to correlate, the soccer ball changes shape, morphing into an American football, and then, as the correlation among the items increases, into a cigar shape. The shape of the scatterplot is highly related to the relative sizes of the eigenvalues; if the eigenvalues are all equal, the shape is a sphere. If the first eigenvalue is much greater than the others, the shape is a cigar, and in such a case the multivariate scatterplot is said to have, essentially, one principal component, or dimension.

In the 10-item example above, the first principal component is said to account for 38.3% of the total variance (or volume) found in the multivariate scatterplot. As the size of the first component comes to dwarf the others, some people say there appears to be but one dimension underlying the items, which, in turn, often leads people to say that the items are "measuring the same thing".

Lertap will also "plot" the item-component correlations (or loadings) in bands. It takes the values found in the p-comps1 row, and makes a little table, such as the one below:

The P-Comp1 bands indicate that there's a group of six items, Q26, Q27, Q29, Q30, Q33, and Q35 with high correlations on the first principal component. If we were to create a new subtest using just these items, chances are very good we'd end up with a coefficient alpha value much higher than that obtained for all ten original items.

And, speaking of alpha values, did you happen to notice that one of the eigenvalues seen above, the first one, has a little black triangle next to it? (This triangle is really red, not black, but for some reason when we took our snapshot of the original screen the color changed.)

When you have your own IStats screen showing, find one of these triangles and let your mouse hover above it. Lertap will display the alpha value for the corresponding principal component; in this case the value turns out to be 0.821 -- it can be shown that this value, 0.821, is the maximum possible value which coefficient alpha could assume for any linear combination of the items comprising the subtest. (Please refer to the technical paper cited below for more information, and also please note that these small triangles will appear only when the corresponding alpha value is equal to or greater than 0.60.)

The Scree Test / Plot

When we think about the first eigenvalue possibly "dwarfing" the others, we might well long for a picture of some type. The scree test was invented by Cattell way back in 1966 to meet these longings. Cattell suggested we graph the eigenvalues from highest to lowest to see if the first eigenvalue, or the first few eigenvalues, dwarf the others. His suggestion remains popular to this day.

We can graph our 10 eigenvalues using a couple of methods. The plot shown below was obtained by selecting the eigenvalues, and then using Excel's Insert / Chart (Line) options. An easier way to accomplish much the same thing is to use an option from the Lertap toolbar: Shorts / Make a line graph (click here for more information on this shortcut method).

The so-called scree test for the number of factors involves nothing more than eye-balling a line graph such as the one above, and deciding where the scree begins. In case you've forgotten, the scree is all the loose rocks at the base of the cliff your friends want to climb, those pesky fallen chunks where your boot will slip in and get stuck, twisting your ankle, granting access to a face-saving retreat to the beer tent in case you were really too chicken to climb the cliff to begin with.

Does the first eigenvalue dwarf the others? Does our scree begin with the 2nd eigenvalue, or the 5th? This question will remain unanswered here; many times the start of the scree is much easier to detect. For references on the scree test, see Catell (1966), Pedhazur and Schmelkin (1991), or search the Internet.

Note that eigenvalues can go negative. This is likely, for example, when SMCs are used on the diagonal of the correlation matrix, when one of the items has no variance, or (especially) when tetrachoric correlations are used. Also note that it is possible for the eigenvalue extraction method used by Lertap to fail; the method is an iterative one which concludes when the iteration process appears to converge. Under some circumstances convergence will not occur -- eigenvalues will not be returned in such cases (but it may be worthwhile to try again, that is, to return to the Run menu, and again request "Output an item scores matrix").

The computation of eigenvalues can be a labour-intensive task for your computer. If you will not be making use of eigenvalues, and have no desire to become an avid scree plotter, then you'll want to turn off the eigenvalue option in the System worksheet (the option's setting is found in Row 22, Column 2 -- set it to "no").

Note December 2004: we received a trial data set from a Lertap user with 150 cognitive items, and 267 test takers. Using this for some new time trials was revealing. It took Lertap a total of 14 minutes to produce its IStats report for this data set (!) -- of this figure, fully 11 minutes were required to extract eigenvalues from the correlation matrix.

More timely comments may be found by paging ahead to the time trials topic.

Related tidbit:

For more about these topics, see "Some observations on the scree plot, and on coefficient alpha", a 16-page Word document with lots of little tables and some wonderful screes, available via the Internet: click here if you're connected.


Factor analysis

As mentioned in the previous topic, when the SMC setting is on, Lertap will output a row with correlations between each item and the first principal factor. Witness:

Lertap's first principal factor is exceedingly simple -- its initial estimate of an item's communality is the same as its final estimate: the item's SMC (no iterations are undertaken to improve on the SMC).

Some users may find Lertap's p-fact1 row to be a useful tool for indicating relative item loadings on the first factor, but those interested in a more complete factor analysis, or, for that matter, a complete principal components analysis, will want to step out to a program such as SPSS or SAS.

How to move beyond Lertap, to get Lertap's item scores into, for example, SPSS?

It's fairly straightforward: the 8-ball 8ball1 on the Lertap toolbar will do the job in good style. Get an introduction to the use of the 8-ball by ... you guessed it!


Creating a text file.

A text file, sometimes referred to as an ASCII file, is a file devoid of special formatting characters. Examples of text files are those which may be processed on a Windows-based computer with the Notepad or WordPad programs, or on a Macintosh with a program such as TextEdit. On Windows computers, text files often have an extension of "TXT".

In the data processing world, data files are often text files (ASCII files), and they frequently have an extension of "DAT". Many of the programs from SSI, Scientific Software International (www.ssicentral.com), and from ASC, Assessment Systems Corporation (www.assess.com), enjoy receiving their input from text files saved with an extension of DAT.

As you now well know, Lertap's repertoire of data processing capabilities includes a provision for creating Excel worksheets formatted so as to be compatible with some of the ASC and SSI programs. For example, Lertap's XCal worksheet is made for use with ASC's XCALIBRE program, while Lertap's DAT worksheet is designed to be friendly to SSI's Bilog-MG program.

But the ASC and SSI programs cannot (yet) read from Excel worksheets. We require a way to save Lertap's XCal and DAT sheets so that they're text files ready for input to the other programs. Can do?

Sure. In fact, there's more than one way. First, make sure that the DAT or XCal worksheet is the active one, the one currently in view. Then use Excel's / Save as TXT (MS-DOS) option. This will create a simple text file with an extension of TXT; this file may then be renamed so as to have an extension of DAT.

Or, here's another way to get the job done: in the process of saving the file as TXT (MS-DOS), using quotation marks around the file name will allow it to be saved directly as a DAT file -- for example:

When saving TXT or DAT files in this manner, Excel is likely to send a message such as this 'un:

Click OK, and be prepared for Excel to say something like this:

This is also okay -- click Yes.

To see if the file you've saved looks okay, get out Notepad, or WordPad, or, for Mac users, TextEdit, and use the File menu to Open your work of art. The screen snippet below shows an example, in this case a Lertap DAT worksheet saved as Ed502.dat, and viewed on a Windows computer with the Notepad program:


Creating a csv file.

A "csv" file is a text file with a certain number of "fields", with each field representing a value of some sort or another.

CSV means comma-separated values. The records (or lines) in a csv file have a series of values (or fields), with commas used to separate them.

Here's an example (two commas with nothing between them corresponds to an empty field):

The csv file above came from a Lertap Stats1b worksheet which looked like this:

How did we get from the Stats1b worksheet to the csv file? We followed a procedure almost identical to that described in the previous topic, "Creating a text file". However, instead of asking Excel to Save as TXT (MS-DOS), we directed it to Save as CSV (Comma delimited) (*.csv).

Lertap users may have a variety of needs which prompt them to save worksheets as csv files. Among these would be a desire to use Lertap's statistics with an item banking and test development system such as FastTEST from ASC, Assessment Systems Corporation (www.assess.com). The latest versions of FastTEST have an Import Wizard which makes it a straightforward matter to pick up values in a csv file. In the example above, we'd tell FastTEST to pick up csv field #7 as the "P-Value", and csv field #8 as the "ItmTtlCorr".

The number of columns seen in a Stats1b report depends on the number of response options, or alternatives, used by a subtest's items. At times there will be too many columns, too many fields when the worksheet is saved as a csv file, for easy use with FastTEST. In this case you'll want to delete some of Stats1b's columns before making the move to save as a csv file.

Is it difficult to delete Stats1b columns? Nope; it's real easy. Use the toolbar's Shorts menu to "Turn row and column headings on/off". Then get out your mouse, and right-click on, say, column 2. Left-click on Delete, and guess what? Bingo! -- the column is gone.

Now, you know how we've been saying there may be too many Stats1b fields, and how you might want to delete some if you're making a csv move to FastTEST? Well, come to think of it, you might want to insert a new column in the Stats1b worksheet before saving it as a csv file. Yes. FastTEST assigns and carries a UniqueID field for each item. Your work might be a bit easier if you inserted a new column in the Stats1b worksheet, and typed each item's FastTEST UniqueID into it before saving as a csv file. This may speed up the task of importing the item stats.

Is it difficult to insert a new column in the Stats1b worksheet? Yep, it's real tough, about as hard as having to quaff a few ice-cold Emu Exports on a hot summer's day. (Be sure to use the Shorts menu to turn column headings on first.)

The item discrimination value seen in Lertap's Stats1b report, "disc.", is a point-biserial correlation coefficient corrected for part-whole inflation. (The manual discusses Lertap's statistics in some detail.) It is possible to get the Stats1b report to include the biserial equivalent, something which is done by turning on Lertap's "Experimental Features" option. Please refer to the following URL for a discussion of these features:

http://www.lertap.curtin.edu.au/Documentation/ExperimentalFeatures.htm


Time trials

The data minuted in the secs-y table below were obtained in October 2003, on a Pentium 4 running at 2 GHz. N is the number of data records in the Data worksheet; Nits is the number of items to be scored; Item scores indicates the amount of time to produce IStats' matrix of item scores; No eigens indicates the total time to completion, that is, time to create the item scores, the matrix of Pearson correlations, the matrix of tetrachoric correlations, and the DAT worksheet; With eigens indicates how much longer the job took when eigenvalues were also computed.

N | Nits | Item scores | No eigens | With eigens
450 | 40 | 13 secs. | 27 secs. | 51 secs.
1,400 | 25 | 24 secs. | 33 secs. | 42 secs.
2,800 | 25 | 48 secs. | 1:02 mins. | 1:11 mins.
5,600 | 25 | 1:45 mins. | 2:12 mins. | 2:21 mins.
11,200 | 25 | 4:04 mins. | 4:51 mins. | 5:08 mins.

Note December 2004: we received a trial data set from a Lertap user with N=267 and Nits=150. It took Lertap a total of 14 minutes to produce its IStats report for this data set (!) -- of this, 11 minutes were required to extract eigenvalues from the correlation matrix. This test used the same computer mentioned above, a Pentium 4 running at 2 GHz. At the end of 2004, this Pentium would be considered as being quite dated (it was new in early 2002); if you're wanting IStats reports for data sets with large Nits, hopefully you'll have a more powerful computer to work on.


Response similarity analysis

Response similarity analysis, RSA, involves getting Lertap to compare all possible pairs of students to see if their item responses might be similar. This sort of analysis is generally undertaken to see if some students might have colluded in creating their answers, something which is generally considered to be "cheating".

If your data set involves "N" students, the number of pairs to be compared will equal (N)(N-1)/2. When N=100, there will be (100)(99)/2 = 4,950 student pairs to compare. When N=5,000 there will be more than twelve million (> 12,000,000) student pairs to compare (!). But fear not: Lertap will crunch your pairs without a whinge, asking only that you muster some patience when N gets over 800 or so (see time trials below).

An RSAdata worksheet forms the base for similarity analyses. RSAdata worksheets are made whenever the "Output item scores matrix" is taken from the Run menu, and the RSA option has been set to "yes" in the System worksheet.

Once an RSAdata worksheet has been created, another option on the Run menu, "Response similarity Analysis (RSA)" will get Lertap to produce its three RSA reports: RSAcases, RSAtable, and RSAsig.

The RSAcases report is the bread and butter of Lertap's RSA analysis. Here's a typical sample:

An RSAcases report presents data for those pairs of students whose item responses have been judged to be "suspect", using criteria developed by Professors Harpp & Hogan.

The item responses given by each pair of students are found under the Responses column, using a format suggested in the "SCheck" program from Wesolowsky (2000): a full stop (or "period") indicates a correct answer. Each of the two students above had 20 correct answers.

The 2 seen at the start of each response string indicates that both students selected "2" as their response to the first item. On this item, both students made an error, failing to find the correct answer. Not only did they both make an error, but they made an identical error on the item. Wherever the student incorrect responses match, they have an "exact error in common". It's pretty easy to see that the two students had nine matching errors, nine "exact errors in common". Over all 30 of their item responses, there were only two response differences.

The values of the Harpp-Hogan measures are found under the EEIC, Index, and Sigma columns of the RSAcases report. Briefly, Harpp-Hogan methods are based on (1), determining EEIC, the number of exact errors found in common in student responses; (2), comparing EEIC to "D", the total number of response differences found, a comparison made by dividing EEIC by D, producing the "Index"; (3), developing a response probability measure for the pair of students, and comparing it to a distribution of similar measures formed from non-suspect pairs. The probability measure is found under the "Log" column, with "Sigma" indicating how significant the pair's probability measure was.

EEIC, Index, and Sigma measures are computed for all possible pairs of students, not just for those whose results come to feature in the RSAcases report. When a pair's EEIC and Index measures are above preset cutoff values, the pair's results are said to be "suspect", meaning that the pattern of their answers to test items was suspiciously similar.

All pairs found to be "suspect" are entered in the RSAcases report. To these the final Harpp-Hogan criterion is applied: if the Sigma measure for a suspect pair is above the preset Sigma cutoff value, the pair's results become "significantly suspect", or "very suspect". Their results receive special highlighting in RSAcases: a pink highlight is added to their ID, EEIC, Index, and Sigma entries. It's these pairs which we might then investigate further. Did they have the opportunity to cheat during the exam? Were they seated close to each other? Were they seen to be using mobile phones, or noted to share the same eraser?

It is the nature of the RSA business to want to have a number representing the extent of possible cheating. In Lertap's RSA analysis, that number corresponds to the number of RSAcases pairs whose entries are "in the pink". When an RSAcases report has more than five entries, a small section at the end of the report summarizes results, as exemplified below:

In this example, the RSAcases report had 57 entries, 57 paired student results. Of these, 35 were "in the pink". We might say that our RSA analysis uncovered 35 pairs whose item responses were "significantly suspect", or, in Wesolowsky's terms, "excessively similar". We can't yet say for sure that they cheated, but we've got reason to question their results.

More than one RSA analysis may be applied to the same RSAdata worksheet. As discussed below, there are several options which control how an RSA analysis runs; it is quite common to specify an analysis which looks not at all students, but only at those whose test scores fall within a certain range. In some cases, we might want to exclude "weak" test items from the analysis. Note the numbering on the reports in the little example shown immediately above. "RSAcases8" would imply that there were at least eight RSA runs used (quite unusual).

The RSAcases report conveys the essence of Lertap's analysis, but two other reports are produced for those who care to delve further into the results.

One of these is RSAtable, exemplified in the screen snapshot below:

An RSAtable report "plots" the values of the Harpp-Hogan Index measure for all those pairs of students having an EEIC value above the preset cutoff. Most H-H Index values will be less than 1.0 in magnitude. To be noted is a special case: the H-H Index is a ratio, one whose denominator, "D", may be zero. When this occurs, Lertap sets H-H Index equal to a value of 999.

Lertap's RSAtable report is made to resemble Figures1, 2, and 3 in Harpp, Hogan, & Jennings (1996).

The RSAtable report is a hold-over from Lertap 5.5 where it was used as the main indicator of potential cheating, a role which has now been assumed by the RSAcases report.

The third Lertap RSA report is RSAsig, a worksheet which contains a wealth of information. RSAsig has three main areas: top, lower-left, and lower-right.

The top of a typical RSAsig report has been captured here. Such reports contain data pertaining to all student pairs whose item responses are not suspect; these are all those pairs with an EEIC value, and/or an Index value less than respective preset cutoff figures.

The entries in the RSAsig report are sorted on column 11, from highest Sigma (H-H sigma) to lowest. The Log(PROB) column, abbreviated as "Log" in RSAcases, is the logarithm of the Harpp-Hogan response probability measure, "PROB", described in Harpp & Hogan (1993).

The lower-left portion of an RSAsig report is shown above. Only one suspect student pair was found in this analysis of 5,565 total student pairings. Thirty (30) items were involved in the analysis, and 106 students. Cutoff figures for the three Harpp-Hogan criteria are shown as "minimum" values under the "Run control" heading. No items were excluded from the analysis, and a score range of 0 to 100 was processed.

Above is a snapshot of the lower-right area of an RSAsig report. The descriptive statistics, from "minimum" to "kurtosis", have to do with the 5,564 Log(PROB) and H-H Sigma values found in rows 3 through 5566 of the worksheet.

The little "expect - found" table is used to gain an idea of how closely the Sigma values found followed those corresponding to the normal curve. Under a normal, or "Gaussian" distribution, 27.20% of all cases will lie between one and two standard deviations on either side of the mean; for the dataset above, 26.69% of actual cases were found in this region, slightly less than expected. It's clear that the results found for this dataset did not identically match what would have been expected under a true normal distribution, but they're perhaps not too bad.

Small triangles to the upper-right of an Excel cell signify that a comment has been attached to the cell. Letting the mouse hover over such a cell will cause the comment to appear, as seen below:

In this case, the comment informs us that seven (7) Sigma values were found to the left of -4 standard deviations, compared to the "0.1763788" values which we would expect to find under a normal curve.

It is possible to get Lertap to graph the Log(PROB) values. Do so by using the histogrammer routine.

The RSAsig report will, at times, differ a bit to the samples seen above. There's a limit to the number of rows an Excel worksheet may have; in Excel 2003, for example, the limit is 65536 rows. Whenever the number of student pairs exceeds 65515, Lertap stops entering results in RSAsig, but continues to compute a subset of the descriptive statistics. It then adds a small table with selected results for all student pairs, as exemplified here:

The little table on the left has Log(PROB) minimum, mean, maximum, s.d., variance, and range data for the 220779 student pairs involved in this analysis. We might now consider the 65515 cases whose statistics are given in the right-most table to be a sample from the whole; comparing the sample Log(PROB) mean and s.d. values (-10.45 and 2.35) to those for the population (-10.50 and 2.36) suggests that the sample data are representative.

To read more about response similarity analysis, be sure to refer to the "Related tidbits" at the end of this topic. Of these, if you have time to read only one, make it "Using Lertap 5.6 to monitor cheating on multiple-choice exams".

Lertap's RSA settings

There's a fair smorgasbord of options which control how Lertap goes about its RSA stuff. Look at the following rows from the System worksheet, as captured in January, 2006.

Should an RSA worksheet be created?

If this option is set to “yes”, Lertap will produce a worksheet called RSAdata1 whenever the “Output item scores matrix” option is selected from Lertap’s Run menu. This is the core worksheet for all of Lertap’s RSA calculations. If Lertap is running in “production mode”, there will be one RSAdata worksheet for each subtest. Once an RSAdata worksheet has been created, the “Response similarity analysis (RSA)” option may be taken from the Run menu. It is this option which produces Lertap’s RSA reports.

Cutoff value for the Harpp-Hogan statistic:

This refers to the H-H index. Harpp and Hogan suggest a minimum of 1.5 for this index.

Minimum EEIC value:

EEIC means “exact errors in common”. The recommended minimum is 8, a value which may be lowered to 6 or 7 whenever the number of test items is less than 40.

Minimum sigma value to be an outlier:

Sigma refers to how far a student pair’s probability measure is from the mean of the distribution of probability measures. Sigma is a z-score. If the probability measures are normally distributed, a z-score of +5.0 or -5.0 more is a very rare outcome indeed—only 0.0000003 of the area under a normal distribution lies beyond a z-score of 5.0. In practical terms, an exam given to three thousand students will produce about five million pairings of students; if the students have not colluded in their item responses, only about two of the student pairs can be expected to have a sigma greater than 5.0, assuming that the distribution of probability measures follows a normal distribution.

Mark all records as pickable for RSA?

This option is, in fact, not yet an option. It may be activated at a future date. In the present version of Lertap, students may be excluded from an RSA analysis by removing the comment (the red triangle) from their RSAdata records; students will also be excluded if their test score does not fall within the range of scores specified by the minimum % and maximum % test score values set in the System worksheet (see immediately below).

Minimum % test score for RSA?

Maximum % test score for RSA?

These two settings determine which students will be included in any RSA analysis. A minimum of 0 (zero) and maximum of 100 will see all students included. Note that experienced users of Harpp Hogan methods will often run several RSA analyses for any given test. They may start with a 0-100 range for these settings, or 30-100, and then reprocess the data with revised settings.

Allow on-the-fly min / max % test score reset?

If this option is set to “yes”, then Lertap will ask you to enter the minimum and maximum % test scores each time you select the “Response similarity analysis (RSA)” option from the Run menu. This completely over-rides the Minimum and Maximum % test score settings in the System worksheet.

Automatically exclude weak items?

For RSA work, “weak items” are those where the number of students selecting the item’s correct answer is less than the number selecting one of the distractors, or less than the number of students who omitted the item. If this option is set to “no”, then Lertap will pause every time it encounters a “weak item”, asking if you’d like to exclude it from the RSA analysis. If the option is set to “yes”, then weak items are automatically excluded. Excluding weak items is strongly recommended; if a test has weak items, the EEIC measure will be inflated, resulting in more “suspects pairs”, that is, more student pairs whose item responses may be judged suspiciously similar (possibly implying cheating). Is it common for tests to have weak items? Yes, it is; difficult items with poorly-functioning distractors will often fall under this definition of a weak item. Note that a “weak item”, in RSA terms, does not necessarily mean a bad item—bad items are, generally, those with a negative discrimination index; it is possible for an item to be weak, in RSA terms, but still have an adequate discrimination figure.

SCheck (Wesolowsky)

The RSA analyses mentioned above all have to do with how Lertap looks at the matter of response similarities. Lertap's procedures are based on those first developed by Harpp & Hogan at McGill University, Canada.

At another Canadian university, Wesolowsky has developed other methods for detecting excessive response similarities. Wesolowsky's SCheck program is based on them. Lertap's RSA procedures will automatically produce a file which will slip right into SCheck -- more about this in steps 2 and 4 below.

Summary of RSA steps

To review, here are the steps required in order to have Lertap do its RSA magic:

1. You have to say "yes" to RSA in the right spot in Lertap's System worksheet. As this topic went to press, the right spot was row 25, column 2.
2. You must go to the Run menu, and click on "Output item scores matrix". This will produce the RSAdata worksheet, and also the SCheckData.DAT file. You'll be able to see the RSAdata worksheet right away as it will form part of your Excel workbook, but the SCheckData.DAT file becomes a separate entity, a file on its own, stored on your computer's hard disk. Where? Well, if you had saved your workbook prior to taking this step, it'll be saved in the same folder as your workbook (otherwise you may have to dig around to find it).
3. Next, back to the Run menu, and a click on "Response similarity analysis" if you want Lertap to make its RSAsig, RSAtable, and RSAcases reports. This option may be selected more than once, each time a new set of reports is created.
4. If you want to use Professor Wesolowsky's SCheck.exe program, start SCheck.exe, and get it to work with the SCheckData.DAT file created by Lertap. Read more about SCheck by clicking here.

Related comments

What about selecting a subset of data records before getting into Lertap's response similarity analysis? For example, what if you wanted to select only those students who took the exam in the Business school's main lecture hall?

Well, you'd use Lertap's *tst card on the CCs worksheet to select the desired records. Of course, you'd have to have a column in the Data worksheet which gives exam location information. Let's say this was column 3, in which case the *tst card might look like this:

*tst c3=(Business)

How about using Lertap's RSA support to simply get an estimate of the similarity problems which may pertain to a large data set? Maybe there's too much data, thousands or tens of thousands of students -- too many -- can we possibly get a random sample to work with? But of course. You'd want to read about Lertap's ability to let you Halve and Hold.

Time trials

Having Lertap do RSA things can take time, as you might expect.

From Lertap's viewpoint, there are usually two things to do: make the RSAdata worksheet, and then, when requested, the RSAsig, RSAtable, and RSAcases worksheets.

Our preliminary tests indicate that it does not take all that much time for Lertap to create the RSAdata worksheet. On a data set with 50 items and 1,400 students, it took some 13 seconds to make RSAdata. Doubling the number of students essentially doubled this figure: 27 seconds to process 2,800 students.

The big crunch comes with RSAsig, RSAtable, and RSAcases. For the data set with 50 items, 1,400 students, EEIC min at 6, and H-H cutoff at 1.00, it took four and half minutes for Lertap to create the three worksheets.

Double the number of students to 2,800, and wowser: twenty-two minutes!

One thing to keep in mind here: it will generally not make much sense to run RSA with data sets housing students from more than one exam venue. Because why? Well, think of what we're trying to figure out: are the item responses from any given pair of students surprisingly similar? If Joe sits the exam in Engineering, and Sally sits the same exam in Commerce, would we want to pose this question? What chance do Joe and Sally have to share exam answers? (Maybe they've got some sort of whiz-bang radio set up which is hidden somewhere in their clothing?)

We might have all test results in one Lertap Data worksheet, true, but when it comes time for RSA we'd probably want to break out records according to their exam venue. Interested in this idea? If yes, back up a few paragraphs and read about *tst c3=(Business). Also see the "Breakout scores by groups" option under the Run menu -- it might also be helpful.

Finally, a closing comment: the literature in this area is interesting, and not ambiguous: make it unnecessary to use RSA software by randomly assigning students to seats in the exam venue, and, if possible, by using different test forms, with item scrambling.

Related tidbits:

For more about these topics, see "Response Similarity Analysis", a 17-page Word document with lots of similar topics, available via the Internet: click here if you're connected.

You'll surely want to take in a journal article submitted for publication in 2006: "Using selected indices to monitor cheating on multiple-choice exams", another Word document, some 15 pages in length. This article mentions other software working in the area of cheating detection, such as Scrutiny!, Integrity, and SCheck.

Then, having looked at the journal article, which was critical of Harpp-Hogan methods, you'll have to take in the best-selling, riveting sequel, a paper which explains how Lertap was modified after Harpp & Hogan revised their original guidelines in response to the journal article. See "Using Lertap 5.6 to monitor cheating on multiple-choice exams".


Breakouts

Breakout scores by groups

Suspected you were heading for a breakdown? Lertap can help: use its "Breakout scores by groups" option to obtain a summary table and graph comparing score results for various groups.

To use this option you will have a column in the Data worksheet which identifies groups.

In the sample above, the columns labeled Degree, Gender, and Entry would be typical examples of columns which carry some sort of group information.

Note: you can change the codes used in columns such as these using the "Recode macro" available via the Move+ Menu. It is also possible to exclude certain cases from the breakouts, such as, for example, cases with missing data. Click here to read more.

Now, say we had a Scores sheet such as the one above. We might want to cross, say, Degree, column 4 in the Data sheet, with SelfReg, column 4 in the Scores sheet.

We zip up to the Run menu, and click on "Breakout scores by groups", asking for Data column 4 to be broken out using Scores column 4. Lertap produces a breakout report, and a corresponding plot:

There can be up to 200 levels in the group column. Values in the column may have any length, and may even be numeric. When there are more than 15 levels, Lertap outputs a line graph instead of a bar graph:

It's possible to change just about everything in Excel charts. Right-click here and there on a chart, and see what happens. Change colors, graph styles, and maybe caffeinated coffee to decaffeinated.

P.S.: we need to whisker something in your ear: an option on the Shorts menu will let you get a boxplot of group results. Give a click about here.

Analysis of variance table

A Breaks report, as seen in worksheets with names such as "Breaks1", "Breaks2", and so on, terminates with "ANOVA", a small analysis of variance table, rather like the one pictured below:

ANOVA tables provide information which may be used to index the extent of group differences. In this regard, perhaps the most critical statistic shown in the table is "eta sqrd.", short for "eta squared". This statistic has a range of 0 (zero) to 1 (one). If the groups differ greatly with regard to the "dependent variable", SelfReg in this case, eta sqrd. will be close to its maximum possible value of 1.00. If there's little difference among the groups, eta sqrd. will be low, as seen in this example where a value of 0.01 has been found.

Eta squared is referred to as an index of "practical significance"; it's also commonly referred to as an "effect size" estimator: the larger eta squared, the greater the differences among the groups. As Pedhazur and Schmelkin (1991) point out, effect size estimators are often interpreted as being measures of "meaningfulness": the greater the effect size, the more meaningful the differences among the groups (for Lertap references, click here).

The F ratio seen in the table is used to test a statistical hypothesis, the so-called "null hypothesis": the average value of the dependent variable, SelfReg, in the populations of people from which our groups have been sampled, is the same: the population groups means are equal (so goes the null hypothesis). The F ratio above, 1.02, results from dividing MS (Between) by MS (Within). To test the null hypot, we used to refer to tables of F values -- these days we can simply ask the computer to see how "significant" the F ratio is. Lertap gets Excel to do this, using Excel's in-built "FDist" function. In our case, FDist says that, were the null hypothesis true, an F Ratio of 1.02 or more would be observed 36.3% of the time, given the sample sizes used in our "study".

If you are familiar with tests of statistical significance, you will know that the usual guidelines suggest that the null hypothesis will be rejected only when we find an F Ratio whose "significance" is .05, .01, or even less. Here our value, referred to as "(sig.)", is .363, well above the .05 level -- if we were really testing the null hypothesis, we would not reject it in this case.

The problem with the F Ratio, and its "significance", is that very small differences in means will sometimes be referred to as being "significant" even when the differences are meaningless; this is prone to happen when sample sizes are large. To circumvent this now well known, widely acknowledged problem, a recommended procedure is to carry along an effect size estimator, such as eta squared: if we find a "significant" F, is it confirmed by a large effect size?

Refer to Thompson (2006), or Pedhazur and Schmelkin (1991), for more readings in this very significant area. Thompson's text is a must-read for those interested in effect-size estimation, and the pitfalls of tests of "significance". (For Lertap references, click here.)


To Halve and Hold

This option is used to create two random samples of data records, dividing a data set into halves on a random basis.

How does it do it? It begins by making a copy of the original Data and CCs worksheets, placing them in a new workbook. For convenience, assume that Excel calls this new workbook "Book1".

Then Halve&Hold uses two standard Excel functions to generate a set of random numbers between 1 and the number of data records in the original Data worksheet, denoted as "ArraySize" below:

Randomize

{... more code ...}

RandomValue = Int((ArraySize * Rnd) + 1)

{... more code ...}

The Randomize function provides a seed to Excel's Rnd routine. It uses the computer's clock to do this, guaranteeing that the random numbers generated will differ each time Halve&Hold is run.

Random numbers are generated until half of the original data records have been fingered (that is, identified). The unfingered records are then deleted from Book1's Data worksheet.

Then another copy of the original Data and CCs worksheets is made, and placed in a second new workbook, which we may call "Book2" for purposes of this discussion.

Next, the data records known to reside in Book1's Data worksheet are deleted from Book2's Data worksheet, and we end up with two essentially random samples of the original data, leaving the original untouched.

When the number of data records in the original Data worksheet is not an even number, Book1 will have one more data record in it than Book2.

How to generate a smaller random sample of data records? Halve&Hold always creates halves, workbooks whose Data worksheets have 50% of the records in the original Data worksheet. To get a sample with 25%, run Halve&Hold again, using one of the 50% samples -- for example, if Book1 contains 50% of the original Data records, run Halve&Hold with Book1 to get two new random samples, each with 25% of the original Data records.

Who uses Halve&Hold? Researchers and teachers, often people who are going on to undertake some sort of IRT analysis. At times one wants to have two samples of the original data; one of these might be used to calibrate an IRT model, with the second sample then used to validate the calibration.

Teachers might use Halve&Hold to demonstrate sampling variance -- how do Lertap's scores and item statistics vary as we compare one of the samples with the other?

Time trials, September 2003, on a Pentium 4 running at 2 GHz: with 3,000 original records, the two halves were created in 18.8 seconds. With a bit over 11,000 original records, the two halves were ready in 4 minutes 18.4 seconds.


Sort A to Z

SortAtoZ This icon is used in conjunction with the Scores worksheet. It permits the information in the Scores sheet to be sorted according to criteria entered by you, the user.

When a sort is requested, Lertap makes a copy of the Scores worksheet, and adds it to the workbook as a new worksheet called Sorted. Then Excel's standard sort criteria box appears, and the stage is set -- sorts are made using this new worksheet.

After a sort has been made, may another sort be requested? Yes. There are a couple of ways to make an additional sort. First, the Sorted worksheet could be further sorted by going directly to Excel's standard toolbar. It has the same SortAtoZ icon, usually showing on a toolbar above the Lertap toolbar. You might want to use Excel Help if you're unsure about using Excel sort.

Another way to sort a second time is to delete the Sorted worksheet, and then use Lertap's SortAtoZ icon again. Or, instead of deleting the Sorted worksheet, it could be renamed, after which Lertap's SortAtoZ icon will be happy to again do its job.

At all times care should be taken to see that the Scores worksheet itself is never sorted. Lertap needs to believe that there's a one-to-one correspondence between the records in the Data worksheet, and the records in the Scores worksheet. This will not be the case if the Scores worksheet is sorted, and it's precisely because of this restriction that Lertap makes the Sorted worksheet for users to work their sorts on.

Note November 2004: When Lertap's SortAtoZ icon is working correctly, the following Excel dialog box will appear:

A problem is that Lertap and Excel do not always work as expected. At times this little dialog box will not appear.

For example, on a Macintosh computer there's no way we can get the box to appear as wanted, and, sometimes, very fast Windows machines will also fail to display it. However, all is not lost: Lertap will have automatically selected the sortable portion of its new worksheet, Sorted, and the dialog box can easily be made to appear by doing one of two things: (1) clicking on the SortAtoZ icon on Excel's menu bar, or (2), using Excel's Data drop-down menu, and then clicking on Sort.

After you've done one sort, you should notice that the sortable area of the worksheet remains selected so that you can undertake another sort if you'd like.

Keep in mind that the 8-ball icon on Lertap's toolbar, TheEightBall, may be used to produce a new worksheet with just the scores, that is, a new worksheet which has only one header row (not the two usually used by Lertap), and no statistics after the last row of scores. Some users may find this the best way to work with their test scores.


Shorts Menu

The Shorts menu is basically a collection of some common Excel actions, especially those frequently used at Lertap Central. All of the actions performed by the Shorts menu's options can also be effected by using Excel's own menus (bar one: Compress HistoL intervals); all we've done is made it easier to apply the Excel options we often find ourselves using.

Page forward to the following topics to have a look at our Shorts.

The door is open: send us a note with your own wish list, mentioning the Shorts you'd like Lertap to have. Send it to: support@lertap.com.


Common Excel shortcuts

As of March 2004, our "common" shortcuts consisted of some really simple actions: turn column and row headers on/off; show/hide gridlines; and change the Excel referencing style from "A1" to "R1C1".

Experiment with these. You can't muck anything up.

... page forward for more Shorts ...


Line grapher

The line grapher is, we reckon, a real handy little option.

Here in the Antipodes, particularly at Lertap Central's home base, there's a great propensity to make line graphs from Lertap reports. For example, we will commonly get into a "Stats1b" report, and graph item difficulties, and also item discrimination coefficients. We'll use the Run menu to "Output an item scores matrix", from which we'll plot such things as item means and variances. And we often go for a scree test by plotting eigenvalues.

The Shorts Menu's option to "Make a line graph" makes it possible to get such plots with ease.

How to use this option? Couldn't be easier: select the cells you want to plot, click on the Shorts menu, then click on "Make a line graph". That's it.

Here's an example -- we wanted a plot of item correlations as found in a "Stats2b" report.

We selected the cells of interest by running our mouse over them (there are 10 "cor." values selected in the screen snapshot seen below):

Then we clicked on Shorts / Make a line graph. This is what we got:

You can modify the resultant Excel chart, the line graph, by using Excel's standard chart options, of which there are many. You can add titles, legends, and change lots off colours ... why, you could probably spend two or three hours enhancing Lertap's initial plot, ending up with a graph which, when pasted into your final report, is bound to bring you great kudos.

One thing that's really neat about these line graphs is that you can see the x and y values associated with any of the line graph's points by just letting your mouse hover right above one of the points. Try it -- you don't need to hold down a mouse button -- just position the mouse pointer on top of a point, and the corresponding x and y values will jump out at you. (Note that this won't work with the graph above as it's just a picture of Excel output, not the real thing.)

What's that you're saying? You'd like to know more about Excel charts? Goodonyou. Use Excel's Help system -- it's got heaps of info. Heaps.

Line graph problems

Users just starting to use Lertap's line grapher shortcut may find that it won't work as advertised here. In the tests we've done to date, the problems which arise have to do with selecting cells.

Cells are selected in normal fashion: by highlighting them with the mouse, or by holding down the Shift key and using the cursor control arrows on the keyboard. The cells are expected to have numbers in them -- however, the first cell selected can have text information, such as a row or column header -- if Excel finds the first selected cell to contain text, it'll use the text as a title for the line graph (this can be real handy).

Lertap will fail to make a line graph if: no cells are selected; only one cell is selected; or if the selected cells do not contain numbers (except for what we've just mentioned: the first selected cell may contain text).


Box and whiskers

Once you've used the Run menu to "Breakout scores by groups", you'll have a Breaks report, a worksheet whose name begins with the word "Breaks".

And, once you have a Breaks report, you can use the Shorts menu's "Make box and whiskers from Breaks" option to get a graph which looks like the following:

The data plotted above are from the "MSLQ" study mentioned in Chapter 9 of your favorite read, the Lertap manual.

Results from one of the MSLQ scales, "SelfReg", have been plotted for three groups of student teachers: ECE (Early Childhood Education), Primary, and Secondary.

The top of each box corresponds to "Q3", the 75th percentile; the bottom of each box corresponds to "Q1", the 25th percentile. The line in the middle of each box represents the position of "Q2", the median, the 50th percentile.

Shouldn't the median be halfway between the 75th and 25th percentiles? If the distribution of scores is symmetric, yes, but otherwise no. ("Otherwise" is the usual case as scores are not often exactly symmetric about the median.)

The lines sprouting from the top and bottom of each box are the "whiskers". The top whisker extends from the 75th percentile, Q3, to the highest score which is not an "outlier". Similarly, the bottom whisker extends from the 25th percentile, Q1, down to the lowest score which is not an "outlier". Outliers are discussed below.

A plot such as the one above has a lot of information. The Primary students tended to have higher SelfReg scores, and the range of their scores was the greatest: the P group's whiskers extend a bit further than those for the other groups.

The highest median score is found in the P group.

The SelfReg scores of the Secondary students are, by and large, the weakest: their median score and their "Q1" score are lowest of the lot.

How it works

The "Box and whiskers" option works by first making a copy of a Breaks report. If you were looking at a Breaks1 report, for example, then you'll get a new report called "Breaks1bw", with "bw" standing for box and whiskers.

This new report will have a section at the bottom with summary score data organised in the fashion shown here:

Boxplots, also known as box-and-whisker plots, are an invention of Tukey (1977). They're a very useful way to visually compare group scores.

Lertap uses Excel's stacked-column chart as the basis for its boxplots. The core information for the chart is found in the three "Pct" rows: 25th Pct, 50th Pct, and 75th Pct.

The length of each whisker is determined by finding the highest and lowest scores in each group which are not "outliers", that is, not extreme scores. Tukey defined the limits for outliers as 1.5 times IQR, the inter-quartile range, Q3-Q1. Looking at the "P" group in the table above, IQR = 62 - 50, or 12. Multiplying this by 1.5 gives 18; scores above Q3 + 18, and below Q1 - 18, are Tukey's outliers. In the case of the P group, any score above 80 (Q3 + 18) and below 32 (Q1 - 18) will be deemed an outlier.

Many boxplot routines, such as that found in SPSS (www.spss.com), indicate the presence of outliers by showing asterisks above and/or below the whiskers. Lertap does something different: if there are outliers, the number of them is shown in the "Lowliers" and "Highliers" rows. The wee table above says (for example), that there are 2 "Lowliers" in the P group. You can see them if you scroll up to the top of the worksheet, as exemplified here:

The scores of 26 and 29 are the two outliers for group P. The effective lowest score for this group is 39, which is 11 points below group P's Q1. These 11 points are the length of the "Bottom whisker" for group P.

A histogram would be another way to look at how outlying a Tukey outlier is. If you switch over to the Breaks1 report, and then take the "Histograms" option, using column 3, the column with group P's scores, you'll see something like the following:

The two Tukey outliers have been highlighted in yellow above, and it's now possible to get another idea of how extreme these scores are: notice the distance between them and the score of 39 (highlighted in orange). These scores are so extreme that they caused the score distribution to have negative skewing (-0.37). However, the boxplot indicates that, once these scores are removed from the scene, the skewing actually appears to be towards the high end of the distribution; this is so as the top whisker in the boxplot is substantially longer than the bottom one.

Note: a common cause of Lertap boxplot failure relates to the codes / names used to label the groups. In this example, the labels are E, P, and S. Failure is likely when the codes are numbers, or single digits, such as 1, 2, 3. And failure is pretty much guaranteed if one of the labels is blank. If you get a message from Lertap or Excel concerning this matter, go back to the corresponding Breaks report, such as Breaks1, and change the codes. Note that it's possible to still use digits if you convert them to text -- an easy way to do this is to simply place an apostrophe before the digits: '1, '2, '3 and so forth.

Flexibility

Once you have one of these boxplots on the screen, it's real easy to alter its appearance. Right-click on the chart, and Excel will open up lots of options. To set the score range used in the plot, right-click on one of the scores seen along the vertical axis, and then take the "Format Axis" options.

Note that you can delete whole columns in the Breaks1bw report, and the corresponding box and whisker set will disappear from the plot. This is useful when you've got too many groups, and want the boxplot to highlight only some of them. For example, if above we eliminated column 2 from the Breaks1bw report, the resultant boxplot would contain just box-whisker sets for the P and S groups.

Related tidbits:

More about the use of Excel stacked column charts for boxplots may be found in work by Jon Peltier (Lertap's boxplots are largely based on Peltier's suggestions):

http://peltiertech.com/Excel/Charts/BoxWhisker.html

Much more about boxplots, or box and whisker plots, may be found on the internet, and in many statistics texts (such as Thompson (2006)).


Compress HistoL

Compress HistoL intervals.

 Return to Introduction

The histogram icon on Lertap's toolbar is used to tabulate the values of a given set of scores. Its main output is what's called a "HistoL histogram". For an example, click here.

It is often the case that a HistoL histogram will have too many intervals. There'll be one interval for each possible score, starting with the minimum score and ranging up to the maximum score. If the minimum score came out to be 4, for example, and the maximum 100, there would be ninety-seven (97) intervals in the HistoL report. Oft times some of the intervals will have no entries; for example, if no-one got a test score of 13, there will nonetheless be an interval in the HistoL report for 13.

The "Compress HistoL intervals" option lets you reduce the number of intervals in a HistoL-type report.

To use it, start by going to a HistoL report, such as, say, "Histo1L".

Then go to the Shorts menu, and take the Compress HistoL intervals option.

Lertap will make a copy of the Histo1L report, calling the copy Histo1LCa. The "LCa" letters mean copy "a" of a compressed L-type histogram (if you use this option again with Histo1L, the next copy will be called Histo1LCb).

You'll then see a little dialog box such as the one seen below:

At this point you'll enter a compression factor, an integer, a number from 2 to 9. Let's see -- Lertaps says that my Histo1L report presently has 94 intervals. Were I to make an Excel chart from the Histo1L report, using all 94 intervals, I know, from experience, that the chart would either be too large, or, if I've re-sized it to make it small enough to print on a standard page size, it'll be too "busy", too hard to read.

So I'll enter 4 in the box, and click on OK.

Before you can recite the birth dates of all the people in your family, Lertap will come forth with a nifty table which looks something like this:

If your luck holds good, there will also be a spiffy little chart, a true wonder, ready for inclusion in that report you wanted to have ready for the school board meeting on Monday:

Now, you can do all sorts of things with both the table and the chart shown above.

If you change the entries in the table, the chart will change too, immediately. If you wanted, you could delete the chart, make some changes to the table, and then take the Shorts menu's Make a histogram chart option to get the chart back. For example, you could maybe insert a new row in the table to indicate a cutoff point, and then make the histogram chart again:

Note that while you were still trying to remember all those birth dates, I got into the chart and fiddled around with it. I changed the title, and the font size of the values along the axes. How'd I get just one bar to be black? By clicking once or twice on the bar, until it was the only selected one. When it had obviously been selected, I then right-clicked on it, and made use of the options which Excel opened for me.

Complete?, you ask. Why is the title on the x-axis "Complete"? Because that's what this test score was called, all the way back in the CCs sheet:

Note that the title of this test, Complete, appears in quotes at the top of the table, in the first row. You can change the title by typing over it, or, more simply, by just changing the title in the chart itself.

If you like this option to compress HistoL intervals, you'll also sure to like the next topic.


Make a histogram chart.

The histogram icon on Lertap's toolbar is used to tabulate the values of a given set of scores. Its main output is what's called a "HistoL histogram", as discussed in the previous topic.

The histogram topic of this document (that is, Lelp) mentions that there will be more output if you've got the Analysis ToolPak Add-In installed. Besides Histo1L, you can expect Lertap to create Histo1E, a special chart made by Excel.

The Histo1L and Histo1E Lertap reports are useful in many cases, for sure. But there have been times when users have pined for enhanced histogramming capabilities.

The Shorts menu's Make a histogram option, introduced in September, 2005, can replace the Histo1E output, making the Analysis ToolPak Add-In redundant.

For examples of the histogram charts created by this new option, please see the previous topic.

This option may be applied whenever you've got a HistoL or HistoLC report opened.

Both of these reports have the same basic format, as exemplified below:

When we say that these reports "have the same basic format", we mean that both have their first two rows reserved for header information, and that both have the crucial data to be graphed, or charted, in the second and third columns, starting in row 3.

Note that both worksheets have "Histo" as part of their name. This is also important: the Make a histogram chart option will work only when a worksheet has "Histo" in its name, has the first two rows reserved for header information, and has the data to be graphed in the second and third columns.

The header information found in the first two rows can be anything. If the text found in the first row has a word with quotes around it, that word will be used as the title for the chart's x axis (otherwise the title will appear as "Scores").

The HistoL and HistoLC reports, or worksheets, are normally created by Lertap. But you may make your own. If you have tabular data of this nature to chart, simply make a new Excel worksheet, enter the data in the second and third columns (starting in row 3), and rename the worksheet so that its name includes "Histo". For example, you could name the new worksheet HistoDatosMios1.

How do the histograms created by this option differ to the Histo1E type of histogram? The most obvious answer would pertain to formatting: the histogram bars seen in Histo1E are light blue in colour, compared to the indigo-blue bars made by this option, and the Histo1E bars are much narrower.

But the most important difference has to do with method. The Histo1E charts are made by an Excel routine included as part of the Analysis ToolPak Add-In, whereas the new HistoL and HistoLC charts are just Excel clustered column charts. These new histogram charts do not require a special add-in -- you could make them yourself if you knew how to use Excel's clustered column chart options.


Macs Menu

Holy macro! (?)

There are quite a number of things to say about the Macs menu, foremost among them: this is a very volatile topic. Many of the screen shots seen in this topic, and those immediately following, might well be out of date when you get around to comparing them with what you might see on the version of Lertap you've got running at the moment.

The reason for this? The macros accessed via the Macs menu are wide open, and subject to change at almost anytime. In fact, when you try the options found under the Macs menu, you may very well end up getting a message such as this:

But let's start at the start. In early September, 2008, the Macs menu looked like this:

Keep in mind that your Macs menu might not look like this, okay?

The names of the options seen under the Macs menu are the names of macros. Macros? Whatsit? If you don't know what a macro is, open Excel Help, and toss it a search term such as "about macros", "using macros", or "creating macros". Macros are based on computer code; they're special-purpose computer programs, often quite brief. Don't know how to program? Not to worry -- you can get Excel to write the code for you by using its macro recorder.

The macros accessed via the Macs menu are stored in a special Excel file called Lertap5MacroSetA.xla. The "xla" extension on this file's name says it's an Excel "Add-In".

Add-In? Whatsit? Ask Excel Help to tell you "About add-in programs".

How to get the Lertap5MacroSetA.xla file? The answer depends on when you installed your version of Lertap.

On Lertaps installed after mid-September, 2008, the Lertap5MacroSetA.xla file was placed in the same directory (or folder) as the Lertap5.xls file. It should be on your computer, perhaps in the directory pictured here:

If you can't find the file in a directory such as C:\ASC\Lertap56, try searching for it. If you can't find it anywhere, then refer to the "Related titbits" below to download it; it's a freebie.

Actually, you don't really need our version of Lertap5MacroSetA.xla. You can make your own. Excel Help says you can learn how to make an add-in by referring to the Microsoft Office XP Developer's Guide. But why bother? Just fix yourself up with a copy of our xla file, Lertap5MacroSetA.xla, and take it apart, modifying it to suit your own needs.

Anyway, once you know that the Lertap5MacroSetA.xla is indeed on your computer, and you know exactly where it is, then you're ready to add it in.

Go to Excel's Tools menu, and click on Add-Ins.

This should bring up the Add-Ins box. You should see something like this:

There are seven Add-Ins available in the list seen above. You may see fewer on your computer.

At this point, you need to click on the Browse button, and navigate to the Lertap5MacroSetA.xla file. On our computer, the browse process lead to the Lertap56 subdirectory (or folder):

Click on the file, then on the OK button. You should see the following:

There is now about a 50-50 chance that the macros in Lertap5MacroSetA.xla will work. Try one. Click on the Macs menu, and then click on, say, MacroMio9:

If Excel says it cannot find the macro, close Excel completely. Then start it up again, or simply start Lertap (which gets Excel to start). Try the Macs menu. If the macros do not work, none of them, then close Excel, open it, and repeat all the steps above. Still no luck? Try writing to us at: support@lertap.com.

There's much more about macros in the following topics. Just page ahead.

Related titbits:

The Lertap5MacroSetA.xla file may be downloaded by visiting: http://www.lertap.curtin.edu.au/Software.htm.
A paper which has more on the development of macros may be admired at: http://www.lertap.curtin.edu.au/Documentation/AngelLearningLertapMacros1.doc (Word file, about 170 KB).


Linking to macros

The previous topic mentioned that the macros activated from Lertap's Macs menu are stored in an add-in file called Lertap5MacroSetA.xla.

The links from Lertap to the macros in Lertap5MacroSetA.xla are made by recording the names of the macros in Lertap's System worksheet. The following screen shot shows how part of the System worksheet looked in early September, 2008:

There is a one-to-one relationship between the macro names seen in the System worksheet and the names seen in Lertap's Macs menu. What's seen in the Macs menu are the macro names found in the System worksheet.

In turn, these names correspond to the names of the macros found in the Lertap5MacroSetA.xla file. So, if a user has placed "MacroMio2" in row 102 of the System worksheet, when the Macs menu is opened, MacroMio2 will be the second entry, as seen up above. When a user clicks on this entry, Lertap looks in the Lertap5MacroSetA.xla file for a macro called MacroMio2. If this macro exists, it starts up. If it doesn't exist, an error message will appear, such as the one shown here:

As mentioned, users may either modify the macros already found in Lertap5MacroSetA.xla file, or create their own, saving them in the same file. You give each macro a unique name, and then place this name in one of the rows in Lertap's System worksheet. The next time you start Lertap, the Macs menu will have been updated. Page forward for an example.


Mac example

Let's say that a user named Lee wants to create a macro which will display the date whenever it is activated.

He opens Excel's Visual Basic Editor, inserts a new module in the Lertap5MacroSetA.xla file, names the module ShowDate, and creates the following little subroutine. He saves the file.

Next, Lee puts the name of his new whiz-bang subroutine into Lertap's System worksheet:

He saves the Lertap5.xls workbook, closes it, and then re-opens it. When Lee taps on the Macs menu, here's what he sees:

With great anticipation, Lee clicks on the ShowDate option, and voila!, look:

So, there you go. With our friend Lee-ding the way, can you now add links to your own macros?

Postscript: your macros do not really have to be related to Lertap. If you keep the Lertap toolbar on screen, the Macs menu will always be available, even though you may not actually be Lertapping.


Macro SAQs

SAQ1:

Q: How many macro links can I have in the System worksheet?

A: Ten (10).

SAQ2:

Q: Do I have to have 10?

A: No. When the Lertap5.xls workbook is opened, it populates the Macs menu by reading down the appropriate lines in the System worksheet; in April 2006 these lines started at row 101, and ended in row 110. Lertap stops reading these lines as soon as it encounters an empty cell in the first column.

SAQ3:

Q: I notice you've left two spaces at the start of each macro name in System worksheet. Why?

A: Just because it makes the System worksheet a little neater. There's no real need for the two spaces.

(Thanks to phantom readers for sending in these Qs. Send yours, and if it's a good one we'll put it up here. Send them to support@lertap.com)


Graphics trio

GraphicsTrio These three icons lead to the creation of a variety of graphical summaries of results. Read on ... the next topics discuss these icons in more detail.


Histogram

Histogrammer

Histogram Lertap's histograms are made from the columns of one of three reports worksheets: Scores, Breaks, and RSAsig.

If you're looking at a Scores, Breaks, or RSAsig report, the histogrammer will swing into action immediately after you've clicked the histogrammer icon. In the case of Scores and Breaks, it will usually ask you to indicate which column you'd like to 'gram (the question is not required when you're on an RSAsig report as in that case there will be only one column, Log(PROB), to plot).

If you click on the histogrammer icon whilst viewing another type of report, such as Stats1b, for example, Lertap will take you to the Scores report, and wait for you to click on the icon again. If you didn't want to plot from Scores, you have the chance to click on Breaks or RSAsig instead, after which you have to click on the histogrammer icon.

The standard Lertap 5 histogram dates back to Lertap 2. Its style is exemplified below:

There's a wealth of information in this old-style histogram; for a discussion, refer to Chapter 10 of the manual (p. 174 in the printed version). Histograms in this old format are saved in worksheets with names such as "Histo1L", "Histo2L", and so on -- the L is used to refer to the Lertap-2 style.

It is entirely, absolutely, 100% possible to get Excel to make a more conventional histogram. If you're on a HistoL report, go to the Shorts menu, and click on the option to "Make a histogram chart".

You should expect to see a "chart" like this:

Scaling an RSAsig HisotL 'gram

Have a gander at the following Log(PLOT) distribution, made from an RSAsig report:

It's often of considerable interest to have a more precise display of what's going on in the left tail of a Log(PROB) plot. To do so, double-click or right-click on one of values on the Frequency axis, and then click on the Scale tab:

Above the Maximum has been set at 50 (the former value was 1200). Click OK, and: You called tails?:

Changing the scale of the (Y) axis has made the left-most Log(PROB) outliers visible. Note that these outliers are also sort of visible in the corresponding HistoL sheet:

Look carefully at the f column above, and you'll spot the outliers below the score of -14. There aren't many, but the presence of just a few can influence the interpretation of an "RSA", response frequency analysis.

Using an Excel Add-In

The fancy "charts" above were made by using the "Make a histogram chart" option from the Shorts menu.

You might be able to get Lertap to make such charts without having to access this option. You could try installing the "Analysis ToolPak".

The screen snapshot below indicates the Excel Add-Ins available in Excel 2002 (the Windows XP version of Excel):

Excel Add-Ins are free, and generally easy to install. For assistance, refer to Excel Help.

Lertap checks to see if you've got the Analysis ToolPak Add-In installed. If you do, and if Lertap and Excel are functioning as they ideally should, then the so-called fancy charts will be produced automatically, at least in theory -- in practice we have had some problems with the Add-In, especially but not only when using a Macintosh computer.

Lertap saves Excel ToolPak histograms in worksheets with names like "Histo1E" and "Histo2E"; the "E" means Excel.

Chapter 10 of the manual has a bit more information on histograms.


Scatterplot

Scatterplotter

Scatterplot This icon makes it possible to compare any two columns from the Scores worksheet in a traditional scatterplot, also know as a scattergraph, or a scattergram. A typical plot is shown below, indicating the relationship between a score called "Comfort" and another score, "Kwldge".

Lertap's scatterplots are Excel charts. They may be extensively reformatted. The axes may be expanded; the colours may be changed; even different symbols may be selected instead of the blue diamonds seen above. How? Double-click or right-click on various areas of the chart when viewing it in its worksheet. Or, look for help on "About charts" by using Excel Help.

Excel has some standard data analysis tools which may be quickly applied to scatterplots. For example, below we show the regression line for Comfort on Knwldge, and we've also put in the value of the squared-multiple correlation coefficient, the "SMC", denoted as R2.

How did we get Excel to put in the regression line, and the SMC? Simple. On the Excel menu bar, we clicked on Chart / Add Trendline, and then selected appropriate options. (Our menu bar looked like the one seen below:)

Lertap places scatterplots in new worksheets, one for each scatterplot. These worksheets have names such as ScatP1, ScatP2, and so forth.

Chapter 10 of the manual has a small section on Scatterplots.


Response charts

Item response charts

ResponseCharts Item response charts are made by clicking on this icon.

Two or three styles of charts are made, depending on the type of Lertap report worksheet active when the icon is clicked on.

How to make a worksheet "active"? Just get it to show -- this is best done by clicking on its tab (for an example, click here).

If a worksheet of the Stats1b style is active, then simple item response charts are made for all the items summarised in the Stats1b report (this applies to all reports of the "b" style, such as Stats2b, Stats3b, and so on). The items may be either cognitive or affective.

If a worksheet of the Stats1ul style is active, then quintile-style plots are made for all the items summarised in the Stats1ul report (this applies to all reports of the "ul" style, such as Stats2b, Stats3b, and so on). These 'quintile' plots can greatly assist with the process of identifying how well cognitive items perform.

Page forward to plot your future.


Unidimensional response charts

ResponseCharts If you have a Stats1b-style report active, clicking on this icon will produce charts such as the one pictured below:

Lertap's item response charts are made by Excel. They may be copied and pasted to any other application, such as Word. They may also be extensively reformatted -- for help on this, please refer to Excel Help.

The response "labels" showing at the bottom of the chart are as found at the top of a Stats"b" worksheet. Here's a snapshot showing the top of a typical Stats"b" worksheet:

These response labels may be changed. If they are, the change will carry through to the item response charts.

For example:

Note how the original response labels of {1 2 3 4 5} have been changed to {SA A N D SD}? Look at what happens when item response charts are requested now:

Lertap places item response charts in new worksheets, having names such as Stats1bCht and Stats2bCht.

Item response charts are briefly discussed in Chapter 10 of the manual (page 172 in the printed manual). An example suggesting how item response charts for different groups of test takers may be obtained is provided in Chapter 8 of the manual (pp. 130-133 in the printed manual).

Note that problems can arise when making these charts -- the number of charts which Excel can make is limited, as is the number of fonts which a workbook may have (the charts sometimes use a variety of fonts). These problems are more likely to appear when quintile plots are made from a Stats1ul report; the following topics provide more details.


Quintile plots

ResponseCharts When a Stats1ul-style report is active, a click or two on this icon will produce charts such as this 'un:

The 'quintile-a' plot pictured above shows how five item options, {A,B,C,D,E}, performed over five groups: 'lower' achievers to 'upper' achievers, with three intermediate levels of achievement. The variable used to index achievement may be internal or external; internal is the default, and is simply the total test score. The number of groups is usually five, but may be less -- it's set in the System worksheet.

The 'other' line generally represents respondents who omitted the item.

Wainer (1989) referred to plots similar to the one above as 'option trace lines'. If you haven't seen these before, have a careful look at their message: the yellow line (with small triangles on it) corresponds to the keyed-correct answer, C. The proportion of people in the lower group who selected this option is 0.11, a figure which steadily increases as we get into higher levels of achievement; by the time we get to the 'upper' group, the proportion able to pick out the correct option has increased to 0.74. The graph clearly shows an (almost) linear relationship between achievement and ability to identify the item's correct answer.

Distractors A and B foil fewer and fewer respondents as achievement level increases. Distractor E is quite popular with the lower achievement groups, falling away in the top (upper) group.

Okay? Great; now let's swap things around a bit, and look at the data from another angle:

This graph, a 'quintile-b' chart, plots achievement groups over response options. It shows, in the case of item A29mc, that the most popular response is in fact C, the correct answer. Distractor A nicely pulls off the three lowest achievement groups, especially the very bottom (lower) one. E is a fairly popular choice for the three middle achievement groups, with more than 40% of the 4th group (the next to lowest group) distracted by it.

Items which are good at discriminating the knows from the know-nots will have a particular quintile-b profile: the trace for the upper group, the "knows", will be the highest peak above the keyed-correct option, and the lowest peak for all the other options. The more it's the highest above the key, and the more it's the lowest everywhere else, the more the item discriminates.

Lertap's 'quintile plots' are of course just Excel charts. As such you can change them easily. A chart may be selected in a number of ways; once selected a great variety of chart options may be played with. Below we've asked Excel to include the actual data from which the 'quintile-a' chart was derived (Excel refers to the table below the graph as the "data table"):

These plots are handy. For some users, they may well be more informative indicators of item functioning than any other Lertap report or graph. How do you get 'em? Simple. First, make sure you've got the number of groups setting in the System worksheet right. You can have from 2 to 5 groups.

Then, make Stats1ul (or Stats2ul, ...) the active worksheet by clicking on it. Next just mouse up to the Lertap toolbar, and click on ResponseCharts.

Lertap will click into action, opening up a new worksheet called either Stats1ulChta, or Stats1ulChtb, depending on a setting in the System worksheet. The default action: the first time you click on the icon, 'quintile-a' plots are made for each item, and placed in Stats1ulChta. The next click creates 'quintile-b' plots, recorded in the Stats1ulChtb worksheet. You can reverse the order via that setting in the System worksheet. To use different colors for the trace lines, see the Chart colors topic.

Note: in February 2005 a couple of important options were added in order to improve the way Lertap creates its quintile plots, and to control for a working Excel memory limitation. It turns out that data sets with a large-ish number of items, say 70 or more, can create Excel havoc when Lertap goes about creating its quintile plots, especially when data tables are requested. If you page ahead a couple of topics, or click here, you'll be able to read all about it.

The Diff and Disc figures seen in the plots are taken from the "b" report which corresponds to the subtest involved. If you're clicking off of a Stats1ul report to get your plots, the Diff and Disc values are taken from lines in the Stats1b report. Note that the Stats1ul report has its very own diff and disc values; these are conventional upper-lower indices; in a way they're inferior to the values found in Stats1b in that they're just based on results from two groups -- the lowers and the uppers -- whereas the corresponding values in Stats1b are based on all test takers. (The Disc figure in the Stats1b report is a corrected point-biserial correlation coefficient.)

SAQ: If I change the number of groups, maybe even to just two, can I still get 'quintile plots', even though I no longer have true achievement quintiles? Sure.

SAQ: And, can I get these marvellous graphs even when my Stats1b report is based on a mastery test? Yes, go for it (Master)!

Related tidbits:

There's a paper, a best seller, which has a lot more to do about using quintile plots: http://www.lertap.curtin.edu.au/Documentation/UsingLertapQuintilePlots.pdf (pdf file, about 400 KB).


EC quintile plots

It is possible to make both quintile-a and quintile-b plots with an external criterion score. When the Run menu's External criterion analysis option is selected, an upper-lower (groups) worksheet is created, assuming the subtest involved is a cognitive one, and the option to create upper-lower analyses has been set to Yes in the System worksheet.

When an external criterion is used, the upper-lower (groups) worksheet produced by Lertap will be called ECStats1ul, or, more generally, ECStatsXul, where "X" corresponds to the subtest involved. If the ECStatsXul worksheet is active, clicking on the charts icon will get the quintile plots rolling.

Now, part of the process of setting up an external-criterion analysis involves selecting a score from the Scores worksheet; in fact, it's this score which defines the external criterion.

Lertap will check the selected score to see if it might correspond to a categorical variable, such as Gender or, perhaps, Region. (Note: the Recode macro available via the Move+ Menu is useful for working with categorical variables. This macro will, for example, allow variables coded with letters to be recoded with corresponding digits.)

If the selected score is found to consist of values in the range 1 to 5, Lertap will define the number of 'upper-lower' groups as equal to the number of different values found. For example, if the selected score has only values of 1 and 2, Lertap will set the number of upper-lower groups to 2. If the selected score has values of 1, 2, 3, and 4 (for example), Lertap will set the number of upper-lower groups to 4. (This action over-rides the number of upper-lower groups setting in the System worksheet.)

Here's a practical example: 288 junior high-school students participated in a test development project which investigated the effects of coaching on test performance. About half of the students sat a practice test before taking the real one. Did this affect their achievement?

Data were entered into a Lertap workbook. One of the columns in the Data worksheet indicated whether or not the student had taken the practice test; this column was called "Practice". Practice=1 indicated the student had not taken the practice test, while a Practice value of 2 indicated that the student had sat the practice test.

Another Data column contained a code for gender, 1 for boys, 2 for girls. Other columns housed the student responses to the 70 test items.

The CCs worksheet was set up to score the 70 test items. The Run menu was then used to Interpret CCs lines, and to produce an Elmillon item analysis.

We used the Move+ menu to copy the Practice column from the Data worksheet to the Scores worksheet. Then we went back to the Run menu and started an External criterion analysis, telling Lertap to use the Scores column with Practice values as the criterion "score".

Lertap dutifully produced two new worksheets, ECStats1f, and ECStats1ul. We had a squiz of the latter -- at the very end we observed this info:

Lertap has denoted the two Practice values as EC=2, the Practice group, and EC=1. It will always do this, that is, always denote the categorical variable as "EC", appending its various values. Had there been four groups, we would have had EC=4, EC=3, EC=2, and EC=1.

Notice the two s.d. values? Zero point zero, and zero point zero. This is correct -- the groups have been defined by a categorical variable; all members of each group have the same "score" on this variable -- there is no variance of scores within the groups, no standard deviation.

We clicked on the charts icon once, and got a new worksheet called ECStats1ulChta. Next, we made the ECStats1ul sheet active (by clicking on its tab), and then clicked on the charts icon again to get ECStats1ulChtb.

What did we find? Most of the initial items had this sort of quintile-a plot:

The plot above has lines which seem to be fairly horizontal -- we'd expect horizontal lines if there were no group differences -- truly horizontal lines indicate that the percentage of people selecting an option is the same in each group, meaning that there are no differences among the groups. (Some readers will recognise this discussion as being similar to that heard when folks sit down with their coffee / tea to yak about "item bias" and DIF, differential item functioning. For further discussion, see, for example, the "Detecting Item Bias" chapter in Crocker and Algina, 1986.)

Above, the proportion in each group selecting each of the four distractors looks to be about the same (except for "other"); the proportion of correct answers (Key= 3) was about the same in each group, perhaps showing a slight practice effect (the line rises somewhat as it moves from left to right).

But now look at item A36mc:

The proportion of people who identified the keyed-correct answer (2) was noticeably higher in the practice group. The lines are no longer all close to horizontal.

There were a few other items with patterns like A36mc's. However, it was a study of the quintile-b plots which highlighted a major message: the practice group stuck it out longer -- they answered more test items; students without practice tended to get bogged down, appearing to run out of time. Look:

"Other" means a student omitted the item. Notice that almost 40% of the no-practice group omitted item A49mc, whereas the omit level in the practice group was below 10%. This pattern set in at item A49mc, and continued to the end of the test without exception. (Well, the gap did narrow somewhat after about the 65th item when the proportion of omits in the practice group began to rise rapidly.)

Of course we didn't need plots such as these to reach this finding. No; the numbers are all there in the source data, in the ECStats1ul worksheet. But you might agree that the plots are more effective in conveying the message. If there's a pattern in the data, plots such as these can help to uncover it.


Quintile options

There are a few options which apply to quintile plots. Almost all of them are controlled by the System worksheet found in the Lertap5.xls workbook. Below we've displayed part of the System worksheet, as seen at February 2005:

The quintile options are set in rows 13 through 16. Two of these four rows, 13 and 14, cover options which are discussed two topics back.

The row 15 option, when set to yes, gets Excel to add a data table at the bottom of each quintile plot. Here's a quintile with table:

The information in the data table is the same as that found in the Stats1ul report for the respective item.

Why would one ever set the data table option to no? Because the tables place demands on Excel. There's a limit to the number of charts which Excel supports; an Excel workbook cannot have an unlimited number of charts or fonts. What's the limit? We're not sure, but it has increased over the versions of Excel, at least for Windows users who've kept their Microsoft Office up to date (Excel being part of Office). Without the data table option on (row 15 = no), and with no other charts in other worksheets, we have obtained more than 100 quintile plots from Lertap; with the data table option on, we have, at times, not been able to get all of the expected quintiles. (We have found that Excel will sometimes say that no more charts can be added to the current workbook, or that no more fonts can be added. These Excel chart / font limitations are covered by Internet-based articles, should you care to use your browser to search for related info.)

So much for the row 15 option. What's this setting in row 16? Mark all items as pickable for quintile plots? To understand this option, have a wee look at the two Stats1ul snapshots below:

Can you spot the difference in these two Stats1ul reports? Pat yourself on the back if you've spotted the little triangles in the top one.

We got our mouse to hover above one of the cells with the triangle, and above you see the result.

If, at this point, we click on the toolbar option to get quintile plots, ResponseCharts, we'll get a quintile plot for those items lucky enough to have a triangle. Should an item not have a triangle, it won't get quintiled, there will be no plot.

Coming back to row 16, if it's set to yes, all items will have triangles. If it's set to no, no item will have the triangle.

You scratching your head? Does this make sense? The triangles are either there, or they're not there, depending on the row 16 setting. If they're there, I'll get a quintile plot for every item. If they're not there, I won't get any quintile plots at all.

Yes. Now we can tell you this: you can get into the Stats1ul report yourself, and either delete the triangles, or add them. How? Hover your mouse over the cell, and right-click on it:

The triangles are Excel's way of indicating that cells have comments. Take the "Delete Comment" option seen above, and the triangle will disappear.

If a cell has no triangle, the drop-down menu seen above will have an option to "Insert Comment".

In this way you can control the items which are to be quintile plotted. The triangle means "mark this item for a quintile". No triangle, no quintile.

A couple of questions arise. "If I'm going to insert a comment, what should it say?" Anything you want. "Love you, Mom", "Don't forget to buy milk", "Hope we whomp the All Blacks", "Pickable for quintiles". The comment can even be blank, that is, as long as Excel still leaves the little triangle in the cell's upper-right corner.

"Well, great, but why have a row-16 setting anyway? If I can insert and delete those little triangles by right-clicking on appropriate cells, why have the row-16 option?" For convenience. If you're going to plot all or nearly all quintiles, having row 16 set to yes saves you the onerous task of inserting lots of triangles. On the other hand, if you're only going to quintile a few items, having row 16 set to no saves the time required to delete a swag of triangles.

And, then, again: "Why should I worry? I love quintiles. Can't get enough of them. Triangles R Me! Row 16 is yes, yes, yes forever." But just you wait: there may come the day when you find Excel refusing to give you all the plots. It's happened to us, and it's a bother -- if you have lots of items, there's a chance that Excel will run out of chart or font memory, and you won't get all the plots. Having the row 15 option set to yes (for data tables) increases the chance of running out of memory. Perhaps keep these comments in mind. Some day those little triangles may turn out to be real handy.

I love quintiles, put yeesch, those colors! They don't look good at all when printed on a black and white printer. Not to worry, help is at hand: see the Chart colors topic.


Chart problems

There are a few problems which can arise when asking Lertap to ask Excel to make charts.

Probably the most common of these has to do with Excel running out of chart and font resources as it goes about making its plots. For comments on this problem, please refer to the topics immediately preceding this one.

At times Excel seems to gather too much speed when making charts, and will forget to apply some of the formatting which Lertap has built in. For example, the name of the item is always supposed to be in bold face, such as Item1, but Excel can, at times forget this (perhaps on days when it's feeling meek, not bold?).

Particularly annoying is the sometimes-noted tendency of Excel to scrunch the plots, to make them squatter than they're meant to be. If you suspect your plots are a bit on the flat side, or the fat side, or the squat side, then do this: make them again, and see if that doesn't fix things.

Keep in mind that you can get into Lertap's charts, and change them to your little heart's content: make new colors, put the legend at the bottom rather than the side, add or changes titles -- a good way to pass a rainy day, or to avoid doing something more urgent but less fun.

(For more about colors, see the following topic.)

Be sure to write to us if you'd like to talk about charts. support@lertap.com is us.


Chart colors

It is a fairly straightforward matter to change Excel's standard color palettes.

If you'd like to change the colors Lertap uses to make its quintile plots, histograms, and group breakout plots, you can. In fact, you can alter Excel's color palettes so that the changes you make automatically apply to all charts in a workbook. Once you've colored things in the way you like, you can then pick up the colors used in one workbook, and carry them over to another workbook.

If you don't have a color printer, you may had the occasion to frown after trying to print the colored charts made by Lertap, another Excel application, or a program such as SPSS. Frown not, fair fellow: use the procedures described here, along with some experimentation, to pick colors which will print well enough, letting you get by until the departmental budget has enough funds to buy a color printer.

Here's what to do:

Windows

Go up to the Excel toolbar. Click on Tools / Options and then click on the Color tab. You should see something similar to the screen snapshot seen below:

The Chart lines colors are used in Lertap's quintile plots.

In a quintile plot, the first color encountered in the series of eight Chart lines boxes is used for the first option trace line. At the moment, the first box has a dark blue color -- this color is generally quite okay. It looks fine on screen, and tends to print well on a black and white printer.

The second color in the series is one which Excel refers to as pink. It's a fairly dark pink, another color which is okay on screen, and rather acceptable when printed in black and white.

The third color, yellow, can be a problem. It may look okay on many computer monitors, but it definitely tends to wash out when sent to a black and white printer. The same can be said, to a lesser extent, of the turquoise used as the fourth color.

To change these colors, click on them, and then on the Modify... button. Pick a new color, and see how you go. Note that the changes you make are "live"; as soon as you've changed a color, and clicked on OK, the new color will appear on respective charts.

The scatterplots made by Lertap use the color sequence seen in Chart lines. The "blips" used in the scatterplot use the first color (dark blue unless you change it).

The blue seen in the bars of the Excel histograms made by Lertap is a default Standard color. It's the sixth box in from the left of the second Standard colors row.

Lertap's PlotBreaks charts have bars which use the seventh color in from the left-hand side of the Chart lines series (looks like teal).

The boxes in Breaksbw box and whiskers plots use another default Excel color -- in this case, the second color in from the left-hand side of the last Standard colors row (looks like some shade of cantaloupe).

To copy the colors set in one Excel workbook to the workbook you're currently working with, choose the workbook having the desired colors in the little Copy colors from box at the lower-left of the screen. Above you see Lertap5.xls displayed in the box, but this is only an example, and probably not a good one as the Lertap5.xls workbook uses Excel's default color assignments.

Macintosh

Mac users access Excel color settings by working from the application toolbar: with Excel the active application, click on Excel / Preferences and then choose the Color option. Selecting colors on a Mac can be great fun: just pick the right crayon from the box, if you please.

(The Mac crayons are an optional display; the default display for the Mac color chooser is the same as that shown above for Windows users: boring, but practical.)


Move+ Menu

The Move+ menu permits columns to be copied from/to Lertap's main information worksheets. A column in the Data worksheet may be copied to the Scores worksheet, or, when it exists, to the DAT worksheet.

The Data worksheet is created by users. The Scores worksheet is created by Lertap when the Elmillon program is run. The DAT worksheet, in turn, is created when the "Output item scores matrix" is selected from the Run menu.

To find out what the Move+ menu's options do, click on the topics shown in the box below:

The manual briefly discusses the Move menu in Chapter 10 (in the printed manual, see p. 173). However, the manual's discussion is limited to the first two Copy options seen above -- the other options were added after the manual was printed.


Copy Data column

This option copies a designated column in the Data worksheet to the Scores worksheet.

The columns of the Scores worksheet may contain only numeric data. Before Lertap will copy a column from the Data sheet to Scores, it makes sure only numbers are found in the column to be copied.

Why will users want to copy columns from Data to Scores? There are a few reasons.

One of the most common reasons is to correlate the values found in a Data column with the values found in a Scores column. For example, the Data worksheet may have a column with SAT test scores; these are to be correlated and scatterplotted with the test scores produced by Elmillon, as found in the Scores worksheet. Copying the respective Data column to the Scores worksheet will automatically correlate the Data column's scores with the other scores made by Elmillon, and open the door to use of the Scatterplot icon on Lertap's toolbar.

Users wanting to carry out external criterion analyses sometimes have entered the criterion scores in one of the columns on the Data worksheet. These scores must be moved over to the Scores worksheet before they may be used as an external criterion.

Users sometimes want use Lertap's Eight Ball to export the information found in the Scores worksheet to other data analysis programs, such as SPSS. Before doing so, they often want to copy over columns from the Data worksheet.

When a Data column is copied to the Scores worksheet, Lertap doesn't know what to put in the MinPos and MaxPos cells after it has been copied. It writes "Unknown" in these cells, leaving it to users to put in proper values. (MinPos and MaxPos values are required by some of Lertap's routines, such as the external criterion routine; if Lertap requires these values, it will ask for them.)


Copy Scores column

This option takes all the scores found in a selected column of the Scores worksheet, and copies them to the first empty column found in the Data worksheet.

There are at least a couple of reasons why users want to do this. First, users may want to export the Data worksheet for use in another application, such as, perhaps, SPSS. However, before doing so, they'd like some of the columns in the Scores worksheet to be appended to the Data worksheet.

At other times, users may want to delete the Scores worksheet, perhaps simply to save disk space. Before doing this they'll sometimes copy one or two of the Scores columns to the Data worksheet.


Copy Data to DAT

Copy a Data column to DAT worksheet

The DAT worksheet is a very special one. It's created when the option to output item scores is taken from the Run menu. The DAT worksheet is most likely to be used in conjunction with the Bilog and Bilog-MG computer programs.

When it's first created, the DAT worksheet contains just two real bits of information: some sort of record ID, and a string of zeros and ones representing item scores.

This is ordinarily sufficient for the Bilog program, but users of Bilog-MG often want to have more data in the DAT file. For example, they might want some sort of group identification code between the ID field and the string of item scores.

If the group identification code has been included in the Data worksheet, it may be copied over to the DAT worksheet using this option. In fact, any column in the Data worksheet may be copied to DAT. As columns are copied over, they're added after the ID field. Lertap makes an attempt to keep track of the format of the DAT records by including a Fortran format statement at the top of the DAT worksheet. This statement will usually not be adequate for Bilog-MG -- it's meant to be used as a guide.

Let's look at an example, a data set having a Data worksheet as captured here:

The Data worksheet has ID information in its second column, some sort of Group code in column 3, and item responses starting in column 4.

There were 25 cognitve items, each scored on a right/wrong basis, with one point for the right answer.

After going through the usual process of using the Run menu to "Interpret CCs lines", then "Elmillon item analysis", then "Output item scores matrix", the DAT worksheet looked like this:

The Move+ menu was then used to "Copy a Data column to the DAT worksheet" as we wanted to bring over the Group code from column 3 of the Data sheet. After doing this, the DAT file changed, as shown below:

Notice how the Fortran format statement has changed? In this case the statement is in fact correct, that is, as far as Bilog-MG is concerned. However, as more columns are copied over from the Data worksheet, the format statement continues to use "I" as a field identifier, which may or may not be correct as far as Bilog-MG is concerned. For example, we brought over a numeric column from the Data worksheet, prompting the DAT file to look like this:

Now the Fortran format statement is no longer correct. What Lertap has called an "I3" field should be "F3" -- in Fortran, "I" is used to denote a field containing an integer; "F" is used to denote a real number with a floating decimal. But this ain't a real problem. When you save the DAT worksheet as a text file, you'll end up deleting the Fortran format statement, and Lertap's minor faux pas will go unnoticed.

How to save the DAT sheet as a text file? Just click here, and read on.


Recode Data column

Recode a Data column

Suppose you had a column in the Data worksheet called "Gender", with entries of F for female, and M for male. Then, suppose that, for some reason, you'd like to instead have a code of 1 for female, and a code of 2 for male.

The recode option would be for you.

Suppose you had a Data worksheet, with a column for "Country", with entries such as NZ, AU, CA, US, DO, and VZ. Then, suppose for some reason you'd like to create a new column, to be called "Language", with NZ AU CA and US to all be coded EN, with DO and VZ to be coded SP. (Apologies to CA residents who speak FR.)

The recode option would do the job for you.

Or, suppose you'd like to get group breakouts, with only CA and US selected. You could use Recode, entering a new code of "Exclude" for all records without CA or US, after which you'd go for those breakouts.

Finally, although you know how to use the *tst card to select only certain Data records, you'd like to just delete Data records with, say, SP in the Language column. The Recode option could do it.

How it works

You go to the Data worksheet, and click on the Recode a Data column option under the Move+ menu.

You indicate the column which has the source data, that is, the column with the codes you want to work with.

The Recode macro copies the column to the far right side of the worksheet. Note the use of the word "copies" -- the macro does not delete or alter the original column in any way.

Then, the macro looks at the first entry in the copied column, that is, in row 3 of the newly-copied column. Let's say it finds a value of NZ. This little snap indicates what next happens:

The macro adds another new column to the worksheet, immediately to the right of the copied column.

It then asks you what NZ should become. You enter EN in the little 'What's new?' box, and click OK. All rows whose entry in the copied column is NZ will then have EN in the adjacent column, that is, in the recoded column.

After this the macro returns to look at the next entry in the copied column. It'll find CA (according to the little snapshot above). You'll be asked what CA is to equate to in the new column. You enter EN.

And so forth. We might end up with rows looking like this:

Got the idea? It's pretty simple (which will reflect poorly on you if you didn't get the idea).

At the end of this process, the Recode macro leaves you with two new columns: the copy of the original column, and a column with the recoded values. It's easy to see if the macro has done what you intended -- just scan down the columns. If the result is not what you had in mind, just delete these two columns, and start again.

On the other hand, if the result is in fact what you wanted, you might then want to delete the column which has the copy of the original column. You don't have to do this, but if you do you'll save a bit of space. There are, after all, only a certain number of columns which a worksheet may have (256 was the limit as this topic went to press).

Astute readers might have their hands up at this point: You said you wanted to have a new column called "Language", but instead you have "Country recoded" at the top of the new column. You haven't finished, have you?

Correct (ho-hum). We're left with the back-breaking task of typing 'Language' into the cell which presently has 'Country recoded'.

If your ultimate objective is to get group breakouts, and you'd like to exclude all records with DO or VZ in the original column, you'd respond thusly:

You don't have to type the whole word; just 'ex' will do. When the breakouts routine runs, it will ignore all rows which have been excluded in this manner.

Similarly, if there are records you want to delete, enter the word 'delete' in the little box, or just 'del', without the apostrophes. Lertap will set about deleting rows from the Data worksheet once the Recode macro has worked completely down the original column.

Please note that Lertap will say No-No! if you're asking for records to be deleted from Data when your workbook also has a Scores worksheet. There's a very critical correspondence between the Data and Scores worksheets, and Lertap tries its best to see that this correspondence is not disturbed.

Herewith all the usual warnings about deleting records from Data: you cannot recover them. Best to make a copy of the workbook before deleting records, something you can do by using the New menu.

If you click on OK without entering anything in the little box, the Recode macro will use whatever value you last entered. This makes it a bit easier to apply the same new code multiple times.

If the original value is blank, or empty, then, to maintain the blank, press your keyboard's space bar once, and then click on OK. Otherwise, if you don't want blanks in the new column, simply enter something in the little box, and, in the blank of an eye ....

Finally: as you may know, Excel has its own recode facilities, and they're quite respectable. If you page forward to the next topic you'll see.


Excel's recoder

The Recode macro described in the preceding topic is not really a recoder. It doesn't alter the contents of the original Data column; instead it copies the indicated Data column, and then lets you create a new column with values, or codes, based on those found in the original column. This is much more along the lines of creating a "new variable", or of "transforming" an original variable to a new one (to use terms which may be familiar to SPSS users).

Excel has Find / Replace options under the Edit menu on its toolbar. This is how these options looked in June 2006 when running Excel 2003 under Windows XP:

The Options>> button is what data recoders will want to use:

To give an example of using Excel to recode a column, let's say that we wanted to change every occurrence of 'F' in a given column to '1'.

To do so, we'd select the column, and then fix up the dialog box so that it looks something like this:

If we dared to click on Replace All, Excel would dutifully find and replace all Fs with 1s in the selected column. We could have selected more than one column -- this is a quick and effective way to truly recode values over a number of columns.

It is very possible to get Excel to do more. We might have a column with numeric values, such as "number of years of test experience", and wish to create a new column with a coded experience letter, such as "H" for high, "M" for medium, "L" for low, and "N" for none.

Take a deep breath and look below at the =IF statement seen in the Formula Bar:

The =IF formula does the recoding for us. It says that, if YrsTest (column AM) has a value greater than 10, then column AN is to have an "H", otherwise, if column AM's value is greater than 5, then column AN is to show "M", otherwise, if column AM's value is over 0 (zero), show "L", else show "N".

(Note that we used Lertap's Shorts menu to "Change the referencing style" so that the column headings are letters, not numbers -- this can often make writing Excel formulas easier.)

This looks very, very IFfy, you say? Well, among all the nice things we might say about Excel, one is that there are lots of resources to turn to when help is needed. You can try Excel's Help, and look up the IF statement (in Excel 2003, Excel Help returned a nifty example on how to use IF to recode numeric test results to letter grades, such as A, B, C, D, F). Or, try the internet. Or the local bookstore (perhaps there's now Recoding Excel Columns for Dummies!). Or even try us at: support@lertap.com .


Apply a formula

Apply a special Scores worksheet formula.

 Return to Introduction

Lertappers sometimes want to create a new score by transforming or combining one or more of the scores found in the Scores worksheet.

For example, let's suppose that a user wanted to apply a linear transformation to one of Scores' scores, of a type commonly found in texts and reference books:

y = mx + b

Here, "x" is a score which already exists, while "m" and "b" are constants. The new score is "y".

Let's suppose that m=10, and b=5, making the equation

y = 10x + 5

Have a squiz now of the workbook below. The original score, "x" in the equation above, is called Test1, found in Scores column 2. Get out your best glasses, and look carefully at the Excel Formula Bar, the one which begins with R3C3, and contains a formula which a user entered: =10*RC[-1]+5.

The RC[-1] is Excel's way of referring to a value found in the same row (R), one column to the left (C[-1]).

Not everyone likes to work with the R1C1 Excel referencing style seen above in the Formula Bar. Many users click on Lertap's Shorts menu to change the referencing style so that columns are labelled with letters, as shown below:

Notice how the formula has changed to =10*B3+5? B3 refers to the cell where "x", the original score, is found.

Both of these formulas (formulae) say the same thing -- they just use different referencing styles (you can pop back and forth between the referencing styles as much as you wish; the Shorts Menu makes it easy to do this).

Okay? We've got a user who wants to make a new score by multiplying the original score, Test1, by 10, and adding 5 to the result.

S/he begins by selecting the cell immediately to the right of the first Test1 score, and then enters the formula by actually typing it in, starting with the equals (=) sign.

After typing the formula, the user presses the <Enter> key, and Excels displays the value of the new score, which in this example is 495.

If this is what's wanted, the user then selects the cell with the new formula by clicking on it, cruises up to the Move+ menu, and clicks on "Apply a special Scores worksheet formula". Your beloved little Lertap then applies the formula to all other original scores, determines the descriptive statistics related to the new score, and updates the correlation matrix found at the bottom of the Scores worksheet, lo:

The label given to the new score, NewScore, may of course be changed.

Don't like the results? Select the new score's column, delete it, and start again.

Want to know more about working with formulas in Excel? Look for assistance in Excel Help (there's lots -- you might start by searching Excel Help for "create a formula").


Update correlations

This option refreshes part of the Scores worksheet -- the part where the correlations appear, which is always at the bottom of the worksheet.

You might want to use this option after you have deleted one of the columns in the Scores worksheet. When a Scores column is deleted, the correlations become messy, with Excel tending to display REF# messages, or something equally ugly. Updating the correlations via this option will clean things up.

Remember that deleting rows in the Scores worksheet is a big no-no. If a row is deleted, Lertap loses track of what's what in terms of the data, and who's who.

You may delete the whole Scores worksheet, and you may delete one or more columns of scores without affecting Lertap's internal data structure. But don't delete any of the rows, okay? Promise?

Note: this option was added August 2003, becoming active in Version 5.25.


The Eight Ball

TheEightBall Lertap's Eight Ball is used to reformat a selected worksheet so that it may be more easily used in another application, such as SPSS.

A worksheet is selected simply by making it active; this is often done by clicking on the worksheet's tab. (Click here for an example.)

The worksheets most often candidates for export are Data, Scores, and IStats. All of these worksheets have two rows for headers at their top. The Eight Ball strips off the uppermost of these two.

The Scores and the IStats worksheets have more extra rows at their bottoms. The Eight Ball sees to it that these rows are also stripped off.

Then, after stripping appropriate rows, the Eight Ball saves a copy of the selected worksheet in Excel4 format. It uses this format as it's a simple one, now a rather old one, a format which other systems generally have little trouble reading.

The name of the new worksheet created by the Eight Ball is always XL4Data.xls.

A fairly extensive discussion of using the Eight Ball may be found in Chapter 9 of the manual (refer to pp. 143-144 if you have the printed manual). Additional mention may be found in Chapter 10 (p. 181 in the printed version).


License Menu

A license, in the case of Lertap 5, means a permit to use the software. Without a license, users can use Lertap 5 for only a certain number of days. For example, the "30 Day Trial Copy" of Lertap 5, available at www.assess.com/software/lertap.htm, will run for 30 days, and then lock. Users must purchase an unlock code in order to continue to use Lertap. The License Menu makes it possible to apply for an Unlock Code.

The License Menu seen on your computer will probably resemble this one:

Click on the menu options shown above for more information.


Display current status

Display current license status

If your computer does not yet have a license to use Lertap, a message similar to the one below will appear when you click on "Display current license status".

On the other hand, if your computer has a valid license to use Lertap 5, you should see a message similar to this one:

Once you have obtained a Lertap 5 license, it is possible to transfer the license to another computer. Information on how to do this may be found under the "Setup the license transfer process" option.


Apply for license

Apply for a license

A license for Lertap 5 is obtained by making application to ASC. Click on the "Apply for a license." option, and a screen similar to the following should appear:

Send an email message to Assessment Systems Corporation with the Session ID and Computer ID codes, and ASC will respond with instructions on what to do next. In some cases the Unlock Code is sent directly, but usually ASC will request an order, with payment, before the Unlock Code is emailed back.

Note that it is also possible to fax the Session ID and Computer ID numbers to ASC; the fax number is +1 651 647-0412 (ASC's office is located in St. Paul, Minnesota, U.S.A.).

CopyIDs

This button shows on the form above. It makes it easier to copy the two ID numbers to an email message.

Please allow from one to five days for ASC to respond to your email or fax.


Unlock Lertap

A message somewhat like the one below may be expected when the "Unlock Lertap" option is chosen:

Click on OK, and a familiar form will reappear (this is the same form used to apply for a license):

The Unlock Code is typed into the waiting box, as seen above (in red). If the code is correct, another message appears:

The Lertap 5 license has now been installed. It may be transferred to another computer, if wanted. Information on how to do this may be found under the "Setup the license transfer process" option.


Setup license transfer

Setup the license transfer process

License holders sometimes wish they could shift their base of operations, getting Lertap to work on another computer without having to purchase another license. Can do? Yes. This topic explains how.

The process of transferring a Lertap 5 license from one computer to another begins on the target computer. That is, the process is started on the computer which is to receive the license. In order for this to be possible the computer which is to receive the license must first be installed with the 30 Day Trial Copy of Lertap (click here for a reminder of how to do this).

Once the target computer has been set up with the 30 Day Trial Copy, clicking on "Setup the license transfer process" will produce a message similar to this one:

At this point, a floppy disk is inserted into Drive A: of the target computer. A file is written to this disk, after which the disk is taken to the other computer, the one from which the license is to be transferred.

The disk is inserted into the other computer's Drive A:, and then the "Finish the license transfer process" option is taken from that computer. Another file is written to the floppy disk, which must then be taken back to the target computer, and once again inserted into its Drive A:. Then, the "Finish the license transfer process" option is taken on the target computer. This completes the license transfer. After this, the original computer no longer has a license.


Finish license transfer

Finish the license transfer process

Please refer to the "Setup the license transfer process" topic.


Advanced Toolbar

Lertap has what's referred to as an "advanced toolbar".

Here's a picture of it, as seen on the Windows version in January, 2005 (the one you see on your computer may differ a bit, that is, if you try to activate it -- please note that Macintosh Excel sometimes has a problem with this toolbar, and may fail to display it):

This toolbar has two special icons immediately to the right of Run. The first one, the magnifying glass, unhides and hides the Sub worksheets. Of course, it's always possible to hide and unhide Excel worksheets; this little button simply makes the job easier.

The next icon to the right, the bell, is used to run Elmillon for just a single subtest. This icon is referred to as the "Liberty Bell" in the manual's Chapter 10. (Elmillon is usually accessed via the Run menu, where it's advertised as "Elmillon item analysis". When you activate Elmillon via the Run menu, all subtests are processed.)

Before leading into why these two new icons are sometimes useful, to some, we should mention that the behaviour of the first icon on the toolbar, the X to the left of the yellow smiley face, is changed from normal. This is the "delete worksheets" icon.

Click here to be reminded of how the X normally behaves. When used from the advanced toolbar, the X does not delete the Freqs, Subs, and Scores worksheets.

We should also mention that the behaviour of the Run menu's "Interpret CCs lines" option is different when it's accessed from the Advanced Toolbar. You have the chance to skip the subtest corresponding to each *col line, or "card", as shown below:

A click on the Cancel button will cause Lertap to skip to the next *col card, without doing any processing.

Please read on into the next topics to get a grasp of why some people use the advanced toolbar (some of the time), and how they get it to show.


Advanced toolbar: why?

Perhaps one of Lertap's greatest strengths lies in its ability to fairly easily attach weights, or points, to each one of an item's responses. The *mws line is the way multiple weights are usually applied; mws stands for "multiple-weights specification". *mws "cards" live on CCs worksheets.

What many people do not realise, no matter how often they pull the Lertap manual down from their bedside table for a relaxing read before sleep, is that it's possible to change item weights without changing the control lines in the CCs worksheet.

Let's say this is what you've done: you've set up a nice CCs worksheet, and used the "Interpret CCs lines" option from the Run menu, after which you run "Elmillon item analysis", again from the Run menu.

You look at the output and realise that there's a need to re-weight some of the items. In fact, your need is so special, you're not even sure how you'd go about creating the *mws lines which seem to be required.

What you need to do is roll up your sleeves, perk some fresh coffee, and get into the depths of a Sub worksheet. You can poke any weights anywhere -- once you have a look at a Sub worksheet, it will (hopefully) be obvious what to do.

Here's a snippet from a Sub worksheet:

What we're looking at here is the weights array corresponding to an item from a subtest having Res=(A,B,C,D,E,F). The item could be an affective one; it could be a cognitive one -- the format of the array is the same. The numbers in the boxes, all 1.00 in this case, correspond to the number of points a person will get if s/he chooses one of the options, A through F.

There are four empty boxes for this item -- they'd also have weights (points) if the item used more response codes.

What's "other"? It's the number of points a person gets if her/his response to the item is not A, B, C, D, E, or F. The person may not have responded at all, in which case there might be a blank in the Data worksheet for this person on this item. Many times scanners will record unanswered items, or "funny" answers, as an asterisk (for example, when a person has shaded in more than one bubble on the answer sheet). Blanks and asterisks are caught by Lertap's other category -- in fact anything which is not one of the six recognised response codes for this item, A through F, will be classed as "other" in Lertap.

Consider again the weights array above. It's essentially saying that a person will get 1.00 points, no matter how s/he responded, or non-responded, to this item. Even unanswers get a point. Crazy? No, not exactly. There are times in the life of test scorers when it's necessary to do this.

Put this "crazy" scoring aside -- it's unusual, yes -- but what we want to point out is that you get can into a Sub worksheet, and poke away at the item weights. Any number, positive or negative, may be placed in any of the weights boxes.

True: *mws cards may be used to accomplish the same thing. In Lertap version 5.25, released 8 August 2003, the power of the *mws card was increased so that it could be used to get weights into the "other" category, as exemplified here:

*mws c12, 1, 1, 1, 1, 1, 1, other=1

An *mws line such as this will result in the same weights array seen in the boxes above. (For more examples of other= in action, use the Index, or Search for other=.)

This being the case, it is nonetheless true that some people have a preference for launching their response weights from within a Sub worksheet. The advanced toolbar is for such users.


Advanced toolbar: how use?

To review, the advanced toolbar has two more icons than the normal toolbar. They appear immediately to the right of the Run menu, as seen here:

The magnifying glass unhides or hides the Sub worksheets. It's called a "toggle"; it's an on-off switch.

The Liberty Bell calls in Elmillon, the item analysis program. To use it you must have a Sub sheet open.

Say, for example, that you've been changing weights in a Sub worksheet called Sub3. You're a good scout -- you've gone to Excel's File menu, and saved Sub3 after making the changes. Now you click on the Liberty Bell. In comes Elmillon; out come your results: a new column is added to the Scores worksheet, and then those lovely Stats reports are created, Stats3f, Stats3b, and, if Sub3 corresponds to a cognitive test, probably Stats3ul too. (If these Stats3 reports existed before, they will be overwritten.)

You bewdy.

You look at the results. Hmmm ... maybe give more points to option D on item 21. You return to Sub3, scroll down to item 21, and increase the number of points corresponding to option D. You save the worksheet. You tickle the Liberty Bell. Another column is added to the Scores worksheet -- you can compare the new scores with the last ones. And once again you get all the Stats3 reports.

Another way to go about this: make a copy of Sub3. Call the copy (say) Sub3B. Make the changes in Sub3B. Maybe even change the Subtest Title at the top so the new score will have a new moniker.

Then, with the Sub3B sheet in focus, ring the Liberty Bell. You'll Scores again (onya!), and now you'll have Stats3Bf, Stats3Bb, and Stats3Bul (?) reports to look at. (The Stats3f, 3b, and 3ul reports from the last run will remain unchanged.)

You bewdy Newk!


Advanced toolbar: how show?

How to get the advanced toolbar to show? Go to the System worksheet. Change the UserLevel setting in row 7 to 2. Save. Close Lertap. Reopen Lertap. The advanced toolbar should be there. Write to us if you have probs: support@lertap.com; ring the bell if you don't.


System Worksheet

The System Worksheet

The System worksheet is one of the worksheets found in the Lertap5.xls workbook -- you can see its tab down towards the bottom of this screen snapshop:

The Comments worksheet is displayed above. To see the System worksheet, just click on its tab.

The System worksheet looked like this as of February, 2005:

If you're looking at this page using an interactive version of Lelp (not a printed PDF version), you should find that some of the rows above may be clicked on -- many of the rows hyperlink to corresponding Lelp topic pages.

If the row of interest to you does not hyperlink, you might try referring to the manual for a suitable discussion; as an example, Row 12, the percentage of N found in upper and lower groups, is not discussed in Lelp: see Chapter 10 of the manual (look at page 166 if you have the good fortune of having a printed copy of the manual).

We used to ship Lertap with the System worksheet hidden in order to discourage less experienced users from fooling around and getting into trouble. Trouble is especially likely to result if the settings in Rows 3, 4, and 5 are altered.

If one of the Present Settings is changed, sometimes the Lertap5.xls workbook has to be closed and then re-opened before the change will be effected. However, numerous settings take effect immediately, and do not require closing Lertap. Among these are the settings seen above in Rows 12 through 17, and in Rows 19 through 24 -- if you alter the column 2 setting in one of these rows, the change will take effect without having to close Lertap and then re-open it. (But don't read too much into this. If, for example, you change the item difficulty calculation method via row 19, the item difficulties you've already obtained will not change until you once again use the Run menu.)

Note 1 inserted May, 2005: a "production mode" capability was added to Lertap, with relevant options set in rows 26 through 29 of the System Worksheet. A simple click here will take to you the corresponding topic. (The row-position of these options changed after the following was inserted.)

Note 2 inserted July, 2005: support for response-similarity checking (is s/he cheating on you?) was added, with relevant options set in rows 25 through 27 of the System Worksheet. A simple click here will take to you the corresponding whatchamadoodle.


Input & Output

Lertap Input & Output

Data analysis systems ask users to do at least three things: input information, describe how the information is to be analysed, and, when ready, signal that the analysis should start.

There are always some constraints on how things are to be done. Lertap 5 is no exception; it uses Excel to accomplish much of its work, but not just any Excel workbook will meet Lertap's requirements.

Lertap wants its Excel workbook to have a worksheet named Data. This is where the answers respondents have given to test or survey items are recorded. Lertap wants its description of how the information is to be analysed to be expressed as lines in another worksheet, one named CCs.

The Data and CCs worksheets are referred to as "primary" worksheets. It's the information from these two worksheets that enables Lertap to go about the business of creating its output: the various reports found in worksheets such as Stats1f, Stats1b, and Scores -- these worksheets are referred to as "secondary" worksheets. Users create the primary worksheets; Lertap makes the secondary ones.

The following topics get into some of the specifics of Lertap's primary and secondary worksheets.

Related tidbits:

For the definition of a Lertap workbook, please click here.

For information on how to create a new Lertap workbook, simply click here and you'll be whisked away to our discussion of the Lertap toolbar's New menu.


Lertap workbook def.

Definition of a Lertap 5 workbook

A Lertap 5 workbook is an Excel workbook with the particular features mentioned here.

A Lertap 5 workbook will always have at least two primary worksheets: one of these is named Data, the other is named CCs.

The Data worksheet's top two rows are reserved for titles. The first row may contain any text; it's used as a means of briefly describing the data found in the worksheet. For example, the first row might say "Data collected 10 March 2003 in Psych 501".

The second row of the Data worksheet contains column headers. For example, if the first column of each data record is some sort of ID field, then the contents of the worksheet cell formed by Row 2, Column 1 might be "ID Number". If responses to the first item are found in column 5 of each data record, then the contents of the worksheet cell formed by Row 2, Column 5 might be "Item5".

Data records begin in Row 3 of the Data worksheet.

If the first column of any row in the Data worksheet is empty, or contains a zero, then that is considered to be the end of data. (Users sometimes use this fact when they're testing their CCs lines to make the test go faster, a blank line may be inserted after, say, 10 data records -- this stops Lertap from reading all the data records.)

If the first column of a row in the CCs worksheet is blank, then that is considered to be the end of the CCs lines -- Lertap will not read beyond this line. (This is useful when a user only wants to have the Freqs worksheet produced -- in this case the first CCs line will be *col, and the second line will be blank -- Lertap will produce its Freqs worksheet, and nothing else.)

Ideally, a Lertap 5 workbook has its default font set to Verdana, with the CCs worksheet being an exception in that it may at times make use of the Courier New font.

Lertap workbooks may have a number of secondary worksheets in addition to the primary ones. Examples of secondary worksheets are Freqs, Stats1f, Stats1b, Scores, and so on. Secondary worksheets are usually the result of applying a Lertap or Excel function; for example, the "Elmillon item analysis" option on the Run menu reads data records from the Data worksheet, a primary worksheet, and produces such secondary worksheets as Stats1f and Stats1b.

(For a related topic, see Deleting secondary worksheets.)

Lertap workbooks may also have other user-created worksheets. For example, in data sets with more than one subtest, or scale, users will sometimes create a codebook worksheet which keeps track of the location of the items comprising the subtests or scales.

Note that a Lertap workbook does not include the Lertap toolbar. It bears mentioning that the Lertap5.xls file is indeed a Lertap workbook, but, if someone uses the term "Lertap workbook", they are not necessarily referring to Lertap5.xls. We could say that the Lertap5.xls workbook is a very special Lertap workbook. Why? Because the Lertap5.xls workbook includes the Lertap toolbar, something no other Lertap workbook will have.


Data sheet

Lertap's Data sheet

The name of the Excel worksheet where data are recorded for Lertap analysis has to be "Data". The first two rows of the Data worksheet are for header information, as described in the definition of a Lertap workbook.

Have a look at the top of a typical Lertap Data sheet:

Row 1 is a general "header", or title, which can contain any information you wish, including nothing at all. Whatever is typed in this row will not appear anywhere else; Lertap doesn't read this row. This row is for your own use -- we use it to provide a brief reminder of the information contained in the Data sheet.

Row 2 also has header information. Each column has been given a header, or label: ID; Position; Experience; Gender; Q1; Q2; and so on.

For your information, the CCs sheet corresponding to this workbook had these two lines:

*col (c5-c64)

*sub aff, res=(1,2,3,4)

Item responses begin in column 5 of the Data sheet, and continue through column 64.

Lertap will use the labels found in row 2, columns 5 through 64, as item IDs. That is, the ID for the first item will be Q1; for the second item Q2; ... and Q60 for the last item (not shown above).

Item IDs can be anything, and in theory can have any length. However, we strongly suggest that items IDs be short -- not greater than 8 characters in length. Valid examples of item IDs: Item1; Preg.2; Soal3; Ques2b; SD204; Likrt17a. Having short item IDs makes parts of Lertap's output easier to read; for example, the Stats1f report has a section which looks like this:

The item IDs play a prominent role in tables such as that seen above; the longer the item IDs, the more cluttered the tables look.

If item labels are not found in row 2 of the Data sheet, Lertap will automatically create item IDs of this sort: Item1, Item2, Item3, ....

If it's desired to include ID information for the respondents, such information may be recorded in any column of the Data worksheet (but: Lertap versions dated before July 2004 have to have the ID in either the first or second column). Lertap will use the IDs to label the scores found in its Scores report, providing the respective column header begins with the letters "ID", or "id". Click here for more about this.

Excel has two reference styles used to refer to the rows and columns of its worksheets. Lertap uses what's called the "R1C1" style. In R1C1 notation, the columns of an Excel worksheet are numbers. In the other style, called the "A1" style, columns are labelled alphabetically.

Excel's default referencing style is A1. When Lertap starts up, it automatically changes this to R1C1. Later, when Lertap is closed, it will set the style back to A1 if that's what was in use before Lertap was started. (The referencing style may be manually set by Excel's Tools / Options / General tab.)

How does Lertap find the end of the Data records? It thinks it's come to the end when it finds a row whose first column is empty, or whose first column contains a blank, or whose first column contains a zero. Because of this, it is generally a good idea to see that the first column of the Data worksheet is used for something other than an item response. We say this as non-responses to items are often recorded as a blank -- try to keep blanks out of the first column.

At Lertap central, when in the process of testing out large new data sets, we often insert a blank row in the Data worksheet after row 52. This effectively fools Lertap into thinking there are just 50 respondents (remember: the first two Data rows are for header information); in turn, this lets us test our CCs cards faster, enabling us to quickly see if we've set up the cards required to get the analyses we wanted.


CCs sheet

Lertap's CCs sheet

The formatting of the CCs worksheet is relatively "ad hoc" when compared to the formatting of the Data worksheet. For example, there's no requirement to have rows with header information.

We almost always include comments in our CCs sheets, rows which remind us what we've done, and when. We sometimes even add some sort of colour coding to our CCs worksheets, as shown below:

The CCs sheet above has four rows with comments: rows 1, 2, 3, and 8. Any row which does not begin with an asterisk is considered to be a comment.

We also like to use a fixed-pitch font, such as Courier New, with the CCs worksheet. This makes the information in the CCs cards line up, as seen above in the *key and *alt cards.

Lertap only looks at the first column of the CCs sheet; any information found in subsequent columns is ignored.

When Lertap finds a CCs row whose first column is empty, it thinks it's come to the end of the CCs lines, and reads no more. This can be handy -- we sometimes enter a single *col card in the CCs worksheet, followed by a blank row. At other times, we'll step into a CCs sheet, and insert a blank row after the *col card. We do this as we know that this will get Lertap to make its Freqs report, but nothing else. Maybe with time you'll come to like the Freqs report as much as we do: it provides a quick, no-frills look at our data. We use it to rapidly get a glimpse of how people answered our questions, and to see if some errors may have arisen whilst processing the data. For example, if the items in our test used the default response codes of (A,B,C,D), we'd be surprised to find Freqs reporting it found an E as one of the item responses.

About here we again insert a critical note regarding response codes (this same message may be found under the Cognitive CCs topic). As all readers know, the Data worksheet contains item responses. In the case of cognitive tests, it is common for item responses to be coded as letters, such as the set {A,B,C,D}. For affective items, {1,2,3,4,5} is a popular response code set. If we look down the columns of the Data worksheet, these response codes are what we see -- "but of course", you might say. And well enough. But: what sometimes happens is that users mis-match the codes found in the Data worksheet with the codes found in a *sub card's Res= declaration. For example, if the Data worksheet shows responses as being from the set {A,B,C,D}, and if Res=(a,b,c,d), there will be a crash -- the Res= declaration is wrong -- the Data worksheet uses upper-case letters.

Now, item responses can be just about anything. The answers to cognitive items can be coded as digits; the answers to affective items may be coded as letters. If letters are used, they may be upper case, or lower case. But in all cases, the Res= declaration has to be "fair dinkum"; if the Data sheet uses lower-case letter, then so must the Res= declaration.

Finally, remember the default Res= assignments. For cognitive tests, the default is Res=(A,B,C,D). For affective tests the default is Res=(1,2,3,4,5). If a *sub card has no Res= declaration on it, these default settings are assumed.


Output

Lertap Output

What are the steps in a Lertap analysis? Create a new Lertap workbook. Enter item responses in the Data worksheet. Make up the "control cards" for the CCs worksheet.

Go to the Run menu on the Lertap toolbar. Choose "Interpret CCs lines".

What happens? Lertap has a squiz of the CCs cards, reading down the rows of the CCs sheet until it encounters a row whose first column is empty. Lertap tries its best to interpret the CCs cards, and, if it thinks all's okay, it then starts to read the records in the Data worksheet, going down the Data rows until it encounters a row whose first column is either empty or contains a zero.

Here's an SAQ: If I make an error in the CCs cards, what will Lertap do? It'll power-down the computer, turn off all the electricity, and dob you in by sending an email to your parents. Well, now ... we've tried to train Lertap so that it's nice, so that it attempts to provide a helpful message as to what the error is, and how it might be fixed. But we know Lertap can always do with more training in this regard -- don't hesitate to write to us at support@lertap.com when you find Lertap has not been nice to you.

But let's imagine that the CCs cards are in order, and that Lertap has gone on to read the responses found in the Data worksheet. What happens?

Lertap starts to makes its secondary worksheets. It actually adds new worksheets to the Lertap workbook. The first one it makes is called "Freqs"; simultaneously, Lertap makes the behind-the-scenes series of "Sub" worksheets. There will be one Sub worksheet for each subtest, that is, for each *col card found in the CCs worksheet.

It then hides the Sub worksheets, brings Freqs to the fore, and announces that it's ready for you to squiz the Freqs. If you're satisfied with the squiz, you return to the Run menu, and select the "Elmillon item analysis" option.

What happens? Quite a bit. Lertap returns to the Data worksheet and reads all its records, extracting the responses corresponding to the first *col card in CCs, forming item statistics, and making a subtest score for each respondent.

It writes the subtest scores "on the fly", that is, as it reads the Data records (you can sometimes see it doing this). Writes them to where? To the "Scores" worksheet, another new sheet which Lertap adds to the workbook.

Then Lertap usually creates its two main reports with item statistics. Each of these reports is a new worksheet. They're called "Stats1f" and "Stats1b", respectfully containing "full" and "brief" item statistics. If the subtest is a cognitive one, Lertap usually creates another new worksheet, "Stats1ul", with upper-lower discrimination and difficulty estimates. (In the process of making the Stats1ul report, Lertap creates a temporary worksheet called "Scratch". It deletes this worksheet on completing the Stats1ul report.)

What happens next? If there's more than one *col card in the CCs worksheet, Lertap repeats this process. Each *col card defines a Lertap subtest. For each and every subtest, Lertap adds a score to the Scores worksheet, and creates the appropriate series of Stats reports.

As to nomenclature, the Stats reports for the first subtest are Stats1f and Stats1b; for the second subtest they're Stats2f and Stats2b; ... and so on.

It is possible to control the number of reports made by Lertap. The Stats1f report is always standard, but the Stats1b and Stats1ul may be turned off. How? By making alterations in lines 9 and 10 of the System worksheet. The computational resources used to make the Stats1ul report are rather extensive; turning off this report will usually save a noticeable amount of processing time, especially when there are more than 500 records in the Data worksheet.

Freqs, Scores, and the suite of Stats worksheets comprise Lertap's main output. But it's possible to get Lertap to deliver more. For example, it'll make histograms, scatterplots, item response charts, a worksheet with item scores, subtest correlation matrices, and even another Stats report, ECStats1f (resulting from an external-criterion analysis).

For detailed information on how to interpret Lertap's output, please peruse the manual.


Item difficulty calculations.

The difficulty of a cognitive item is traditionally defined as the proportion of people who answered the item correctly. If, for example, 80% of test takers identified the correct option to Item 1, we'd say Item 1's difficulty was 0.80.

But what if there is more than one right answer to Item 1? What then? What do we do when the scoring of a cognitive item is no longer dichotomous, right/wrong, but instead exhibits polychotomous (or polytomous) scoring?

Well, what Lertap used to do, prior to July 2003, was simply define item difficulty as the proportion of people who selected one of the item's correct answers. By "correct answer", Lertap meant an item response whose corresponding weight was greater than zero. Lertap didn't care if the weight was half a point, a whole point, or 525 points -- any response with a weight above zero was counted as a "correct answer".

As an example, suppose Item 1 had three possible responses, {A, B, C}, with corresponding weights of {0.50, 0.50, 0.00}, and response frequencies of {20, 25, 30}. Then Lertap would say the item's difficulty was (20 + 25)/75, or 0.60.

This way of determining item difficulty in the case of more than one correct answer is still well regarded at Lertap Central. But we agree that this isn't the only way to go about indexing item difficulty in this case, and, in response to a user request, we experimented with making Lertap more flexible. There are now three ways which may be used to reflect a cognitive item's difficulty, and we have changed Lertap's default method to one of the "new" ways.

1) proportional

This is the method described above. Under this method, item difficulty is the sum of the people who selected one of the correct answers, divided by the total number of people responding. This method counts any response as being correct if its corresponding weight is greater than zero. This method does not take into account any differences which may exist among response weights.

2) item mean

A second way of assessing the difficulty of a cognitive item is to simply use the item's average, its mean. If an item has just one correct answer, and if the weight for that answer is 1.00, then the item's mean will be identical to the proportional index of difficulty defined above.

3) item mean / max. weight (default)

Item means can be greater than zero. Traditionally, item difficulty has been measured on a scale which goes from 0.00 to 1.00; if we divide the item mean by the greatest response weight, we effectively re-scale the mean so that it falls back to the 0.00 to 1.00 range. This method of indexing item difficulty does exactly that. When there's only one correct answer to an item, it yields the same result as 1) above.

As indicated, Lertap's default method is 3), item mean divided by the maximum response weight. To change it to one of the other methods, do this: (1) make a change in Row 19 of the System worksheet in the Lertap5.xls file; (2) save and close the Lertap5.xls file.

Finally, we should mention where cognitive item difficulties are displayed. They're shown in the item difficulty bands found towards the bottom of the Stats1f report, and they have their very own column in the Stats1b report.

When the item difficulty calculation method has been set to 2) above, Lertap's item difficulty bands can come under stress since they use a 0.00 to 1.00 scale. In this case, Lertap momentarily pops into the 3) method, re-scaling the mean so that it will fall into one of the bands. However, the item mean will display correctly in the Stats1b report.


Stats1b plot

In November 2006 a new Excel chart was appended to the bottom of Statsb reports for cognitive subtests.

These plots of item difficulty by discrimination are made using the "diff." and "disc." figures displayed in a Statsb report.

The item labels in the case above were Qs followed by an item number; Lertap uses whatever labels are found in the first column of the Statsb report -- when these labels are longer than four characters, Lertap will use only the last four. For example, if an item label is Ques107, the plot will show s107.

Displaying labels will often make the plot crowded, and a bit hard to read. Fortunately, the labels may be easily removed. To do so, right-click on any one of the labels, and an option to "Clear" the labels will appear.

This is what the plot looks like after the item labels have been cleared:

These plots are simply Excel charts; you can reformat them in many ways -- right-click or double left-click at various spots, and options will appear. For example, right clicking in the center of the plot will give access to "Chart Options". One of these options provides the chance to add gridlines to the plot area:


Conditional SEMs

The calculation of CSEMs, conditional standard error of measurement estimates, was added in October 2006 (Lertap version 5.6.3).

Your read of the Lertap manual will have made you full-bottle on the usual standard error of measurement, SEM, and the invaluable role it has to play in the interpretation of cognitive test scores.

It has long been known that standard errors of measurement vary by ability. As we go from test takers with little proficiency, to those with average proficiency, and then on to the strongest students, respective SEMs change. The SEM commonly used is but an average figure, and, until now, it was the only measurement error estimate provided by Lertap.

No longer. Lertap now computes SEMs for various score levels using methods from a paper by Lord (1984). More exactly, Lertap employs Lord's Method III, the binomial error model, and also the adjustment to Method III estimates, known as Method IV.

Have a look at an example of the chart Lertap outputs for CSEMs:

Charts such as this one will be found on worksheets with catchy labels of "csem1", "csem2", and so on. They are automatically computed whenever a cognitive subtest is found to have used number-right scoring, which is the most common type of scoring.

There are always three lines in these graphs. The flat one, the horizontal one, corresponds to the usual SEM value, as seen in Lertap's Statsf reports. The SEM value is a constant; it's the same for all test takers, no matter their ability.

The top line, a curve, corresponds to standard errors computed using the binomial error model, labeled as Method III in Lord (1984). The lower curve is what results when Method III estimates are adjusted using Method IV; Method IV estimates will be lower than Method III's whenever the items used in the subtest have different difficulties ("diff" values, to use the parlance of Lertap's Statsb reports).

So: what's the graph tell us? In this case, results are from a 20-item subtest, one of three subtests used by a university faculty of engineering to screen first-year applicants.

Consider those applicants with a true number-right score of 10 (50% of the maximum possible subtest score). The usual SEM at this score level was 1.96, as it was for all other levels. But the conditional standard errors of measurement at this point were greater: 2.12 using Method IV, and 2.29 using Method III; these CSEMs are a better reflection of the true state of affairs in the center of the score range -- errors of measurement are highest here, falling away as we move left or right to the extremes.

CSEMs will have special relevance when cut scores are used, for example, as in setting pass-fail points. For the subtest pictured above, we'd want to acknowledge greater measurement imprecision for scores near the middle, while at the same time having reason to believe that we have less error when it comes to testing the best students.

Related tidbits:

There's a paper, another best seller, which has lots more about using cut scores, with several examples: http://www.lertap.curtin.edu.au/Documentation/JERM2007d.doc (Word file, about 300 KB).


Import & Export

Lertap is an Excel application. Its input and output "files" are standard Excel worksheets nested within a standard Excel workbook.

We have found that users will often have an Excel worksheet with data which they'd like to use with Lertap. If they rename the worksheet to Data, and add another sheet called CCs, will Lertap work?

Yes. And no. Lertap will certainly work, but its output may be poorly formatted and difficult to read. There are often font problems. Lertap has a preferred font: Verdana. If the user's workbook is based on a different font, Lertap's output may be adversely affected.

When such problems arise, we suggest this: use Lertap's New menu to make a new blank Lertap workbook. Then, copy all of the data records from the original workbook, and paste them into Lertap's Data worksheet. This generally works without problem.

We have seen numerous "text" files imported to Lertap without problem. For a fairly thorough example, we recommend a visit to this URL:

http://www.lertap.curtin.edu.au/Documentation/Samples/MondatY/ProcessingMondaty.htm

Not long ago we happened to be in Central Java, Indonesia, where we saw some 20,000 provincial high school test results imported to Lertap from a "dBASE" file set up by a scanner. The dBASE file had three tab-delimited fields: record number, student ID code, and a string of 80 item responses. A straightforward copy-and-paste from the dBASE file to a Lertap DATA worksheet, followed by application of "The Spreader" in the Data worksheet's third column, quickly set up the data for Lertap processing. (It took about 18 minutes for The Spreader to work through all the data records.)

Lertap has a special ability to import the text (DAT) files used by the ITEMAN program. This could well serve as a general means of importing data prepared by a scanner. Read about it in the next topic.

Are you aware of Excel's ability to dissect a text file? Excel has a "text import wizard", a useful tool which can be of real help when you've got to take apart a text file (the "DAT" files output from some scanners are usually simple text files).

This wizard can be whizzed up in a couple of ways. One way is to use Excel's File / Open options, and under Files of type: ask for "All Files (*.*)", then browse to the file you've got in mind. Or, use Excel's Data / Import External Data, and follow the same steps. If the file you point to is purely text, Excel opens a dialog box which allows the file's contents to be cut out, field by field, and placed in the columns of a new Excel worksheet.

Here's a picture of Excel's text importer in action:

Doesn't the Excel Import Wizard look pretty useful? It is, but beware: it's got a problem. Yes. If there's a string of item responses to be imported, watch out. You have to make sure that the string has no blanks at the start.

Have a look at this screen shot:

Look at row 22 above. It has four blanks at the beginning, probably signifying unanswered questions. When the wizard is asked to import the string of responses, chances are real good it'll simply lop off those four blanks, shifting the string to the left. This is real bad -- this record's test score is going to be wrong.

How to control for this problem? Well, if the file has been created by a scanner, see if the scanner can't be coaxed into saving its data in an Excel-ready format, such as, perhaps, a "csv" file (comma-separated values). Such files come into Excel without having a need to be converted. Another useful format is the trusty old dBASE one mentioned above.

If you want to talk to us about this type of problem, just zip off an email to support@lertap.com. We'll get back to you as soon as we're in from fishing.

While talking about strings of item responses, don't forget about "The Spreader". It's tailor-made to take strings of responses apart, and it loves to be put to work.

As to exporting Lertap worksheets, making them ready for use in another package, please refer to "The Eight Ball", and to the discussion related to having Lertap make its own DAT file.


ITEMAN

ITEMAN is another classical item and test analysis system, created by David Weiss way back when.

When, exactly? As it happens, about the same time as the first version of Lertap: late 1960s.

These days ITEMAN is distributed by Assessment Systems Corporation, www.assess.com. Professor Weiss lists it on the ASC as the 'world's most popular' item analysis program. There can be little doubt that it has developed a real following over the years.

Times might be a'changin, but ITEMAN is still frequently used. It has what some might term a rather dated interface. It produces output designed for a printer rather than for on-screen browsing. Its graphics are almost non existent.

But it's a workhorse, and not difficult to use, especially for sites where test results are generally processed with an optical scanner.

Lertap 5 is able to import files made for ITEMAN, and it does so with real ease. This could be a useful and quick way to import data from a scanner. To use the ITEMAN importer in this manner, you'd first want to download the ITEMAN user manual from www.assess.com and get an understanding of the four control lines which ITEMAN wants ahead of the actual data (pretty easy to set up -- the manual is good).

Access to Lertap's ITEMAN importer is via the Macs menu.

Related titbits:

A paper with more about ITEMAN and Lertap is here: http://www.lertap.curtin.edu.au/Documentation/ItemanAndLertap5.pdf (PDF file, about 1.5 MB).

A discussion of the correlation methods commonly found in item analysis programs, with emphasis on why Lertap's results sometimes appear at variance: http://www.lertap.curtin.edu.au/Documentation/ItemCriterionCorrelations1.doc (Word file, about 190 KB).


Revisions

The manual was printed 1 December 2000. Numerous changes have been made to Lertap since then, some minor, some more substantial.

For a complete record of changes, please refer to the UpdatesSummary page on the website. Its URL is:

http://www.lertap.curtin.edu.au/Documentation/UpdatesSummary.htm

Most of the more salient changes are mentioned in the following topics.

May 2003: added "Lelp" (version change to 5.2)

June 2003: added an item zapper

July 2003: changed item difficulty calculations

August 2003: what you weighted for (other=) (version change to 5.25)

September 2003: to halve and hold forever; Bilog-MG; tetrachorics

October 2003: eigenvalues & SMCs (version change to 5.3)

November 2003: smiles for quintiles!

February 2004: nothing sword-id about this (XCALIBRE support)

April 2004: consolidation (version change to 5.4)

July 2004: record IDs, & formula scores

September 2004: MDO now means: Missing Data Out!

October 2004: enhanced IStats report (version change to 5.4.5)

November 2004: *exc, a new CCs "card" to exclude items

February 2005: three enhancements (EMQs; MDO; quintile options)

May 2005: production mode added (version change to 5.4.6)

July 2005: beat the cheat? (version change to 5.5)

September 2005: histogram charts anyone?

January 2006: beat the cheat 2 (version change to 5.6)

March 2006: more MDO stuff, plus a did-not-see option

April 2006: let us Mac your day (macros R U!) (version change to 5.6.2)

June 2006: a new recoder, and analysis of variance

October 2006: conditional standard errors of measurement now computed (version change to 5.6.3)

(Insert time gap of many months while the Excel 2007 version of Lertap was created.)

September 2008: added a macro to import ITEMAN files

July 2009: improved the boxplot routine (version change to 5.6.4)


Jun 2009 (Ver. 5.6.4)

The boxplot routine for enhancing a "Breakout scores by groups" analysis is now based on an Excel stacked-column chart (instead of a Volume-Open-High-Low-Close Stock chart), resulting in a much superior plot.


Oct 2006 (Ver. 5.6.3)

Conditional standard errors of measurement are now estimated for cognitive subtests. Read about CSEMs with a click here.

A new supporting document related to the use of CSEMs, mastery tests, and cut scores is available as a Word document (about 300 KB).

A scatterplot of item difficulty by discrimination now appears at the bottom of every Statsb report. With a wee click here you'll see great some examples.


Jun 2006

June 2006

June 2006: boys will be girls, and at variance?

Recodes, recodes, recodes: a new option on the Move+ Menu will be useful when you need to change boys into girls, cities into countries, and all such. A click here will show how.

The breakouts report has been enhanced; it now features an analysis of variance table at the end. Have a look.


Apr 2006 (Ver. 5.6.2)

April 2006: how about a Big Mac?

No doubt you've overheard people talking about the new macros they've made to customize their copy of Lertap?

It's true, too: you can now get Lertap to link to your own macros.

Read all about it: the Macs Menu.


Mar 2006

March 2006: adjustments to the MDO, and support for "did-not-see" cases

Pairwise exclusions now apply to the calculation of cognitive item correlations when MDO is used on the *sub card: all item-criterion correlations are now corrected for missing data.

A did-not-see option has been added to the System worksheet. This option works in a manner analogous to MDO: if a person has not been presented with the chance to answer an item (for whatever reason), the calculation of item statistics is adjusted accordingly. A gentle click here will let you see more.

Both of these adjustments will be of use when students see different test items. In some current online testing systems (for example), items are sampled from an item bank, with each student getting a subset of items. Not only do students see a sample of test items, but the items included in the sample will vary from student to student, effectively presenting each student with a different test.


Jan 2006 (Ver. 5.6)

January 2006: four enhancements (version change to 5.6)

While most were out celebrating the arrival of yet another new year, there was no rest at Lertap central. Some substantial changes have been made.

Response similarity analysis, RSA

Enhanced work in this area started July last year. Now Lertap produces three reports for looking at the matter of suspect cheating, with an important probability index, "Sigma", added to help put the heat on a cheat. To find out more you'd want your mouse to nibble here.

Lertap breaks down

What's this? A break down? Nope; just had to get your attention. There's now a new option on the Run menu which will break out results by groups.

Say people have sat your test on chemistry at five different campuses of your university. You have coded test venue into one of the columns on the Data sheet. The new option to "Breakout scores by groups" will quickly produce a "Breaks" table with test results organized by group levels, and a spiffy graph to match, "PlotBreaks".

[****]

Read more. Plus: have a look at growing whiskers immediately below.

Box your whiskers?

The Shorts menu also sports a new option: "Make box and whiskers from Breaks." Once you've got one of the spiffy new Breaks tables mentioned in the previous paragraph, this option will make a copy of the table, and reformat it so that it will suit one of Excel's built-in options for plotting the performance of your stocks.

Don't have any stocks? No matter. Excel doesn't really realize what it's plotting; the "stock performance" graph which results comes usefully close to being a real boxplot, and it certainly has whiskers.

Burma shave?

The histogrammer now uses improved grammer

The good old line-printer-compatible histogrammer from Lertap 2 days, a favorite of many (well, at least some), is now smarter, being capable of plotting the scores found on three different Lertap reports sheets: Scores, Breaks, and RSAsig. Have a squiz.

Use this with the Shorts menu option to "Make a histogram chart", an option introduced Sep 05, and you'll wow your audience for sure.


Sep 2005

September 2005: a new way to make histograms

Two options were added to the Shorts menu, making it possible to change the number of bars a histogram has, and enabling the creation of histograms without requiring the Analysis ToolPak Add-In. The new histograms, referred to as "histogram charts", are easier to modify. Read all about it.


July 2005 (Ver. 5.5)

July 2005: response similarity analysis added (version change to 5.5)

Added support for those interested in investigating whether or not answer copying or sharing may have taken place during an examination.

Two new resources are available under the general rubric of "RSA", response similarity analysis.

As usual, to find out more you'd want to caress your little mouse, and ask it if it wouldn't mind clicking here.


May 2005

A production mode capability was added, making it possible to roll right through, non-stop, from the Run menu's two main options: "Interpret CCs lines", and "Elmillon item analysis".

Settings in this mode also make it possible to roll further, automatically getting histograms, response charts (such as quintile plots), and an item scores matrix (IStats), all without having to wear out your mouse with extra clicks on toolbar options.

Read more about it.


Feb 2005

February 2005: three enhancements

EMQs may now be processed with Lertap. EMQs are extended-matching questions, also known as EMIs (where the I means "item"). A test which makes use of EMQs will typically start out by presenting a series of options, up to 26 of them -- these turn out to be the same as the alternatives, or options, commonly used by MCQs (multiple-choice questions). What makes EMQs different from MCQs? Well, firstly, EMQs use many more options than MCQs. A typical MCQ will have what? Four options? Five? EMQs will have anywhere from 10 to 26.

Then, secondly, EMQ-using tests will have several items which use the same set of options. (This is why the set of options appears before the questions which use them.)

Who uses EMQs? They're popular with the National Board of Medical Examiners in the United States, and they are used in exams created by the United Kingdom's "PLAB", the Professional and Linguistic Assessments Board. We know that EMQs are also frequently used in the Caribbean, particularly in Trinidad and Tobago.

In practical terms, this means that Lertap's Res= declaration may now up to 26 entries; here's a little example, showing an Res= declaration which sets out 20 options:

*Sub Res=(A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T), Title=(EMQs!)

In order to accommodate the use of EMQs, we've changed the way Lertap's *alt card works. It used to be that the entries on the *alt card indicated how many of the Res= characters were used by an item; now the entries actually indicate which Res= character is the last one used by an item. (See example C7 under the Cognitive CCs topic, and remember: *alt is used only when some items use more options than others; if all items use the same number of options, *alt is not needed.)

The second enhancement? New options which control how quintile plots plot. You can have Excel automatically attach data tables to the plots (if you want), and you can tell Lertap to tell Excel that you only want certain items to be quintiled, not all of them. To read about this, you will want to click here.

And finally, the third enhancement concerns adjusting the difficulty index for cognitive items so that unanswered or omitted questions are omitted from the calculation of the index. If you would not like to read about this enhancement, ignore the temptation to click here.


Nov 2004

November 2004: *exc, a new CCs "card" to exclude items

Ways to quickly remove an item from its subtest is a matter discussed in a topic on its own; there's even a paper on the website which deals with the issue in somewhat extensive detail (a link to this paper is found at the end of the remove-an-item topic).

Now removers of items have a new tool: the *exc card, or line, which will probably be the easiest way yet to see that an item, or items, is/are quickly excluded from a subtest. Click here to read about this new method.


Oct 2004 (Ver. 5.4.5)

October 2004: enhanced IStats report (version change to 5.4.5)

More information has been packed into the "IStats" report. Its SMC values are now plotted in a series of ten bands, easing the task of determining the extent to which any single item relates to the others.

The first principal component, or the first principal factor, of the correlation matrix is now extracted, and item-component correlations are displayed in two ways: as a conventional row of values, and in the ten-bands format similar to that used for the SMC coefficients. A technical paper was added to the website to demonstrate how to interpret this new output. Read more about this in the eigenvalues topic.


Sep 2004

September 2004: MDO now means: Missing Data Out!

The power of the MDO option has been increased. Including MDO on an affective subtest's *sub line now gets Lertap to correct its brief stats report (such as Stats1b) for unequal response n's. Items with missing data may now have their statistics adjusted so that they're based only on valid item responses.

This is likely to be a handy revision for survey users. Read more about it by clicking here.

As part of this revision, we modified the CCs lines corresponding to the Lertap Quiz so that they show off the functioning of the new MDO whenever a user takes the cook's tour.


Jul 2004

July 2004: record IDs, & formula scores:

Prior to this revision, data records with ID information had to satisfy two criteria: the ID itself had to reside in either the first or second column of the Data worksheet, and the column header used for the ID field had to begin with the letters ID, or id, or Id, or iD.

Now the first of these requirements is gone. Zapped. The column with ID information may now be any column. This change, prompted by a request from Barbara Foster, University of Texas Southwestern Medical Center, will be welcomed by those who like to put the first item response in the Data worksheet's first column.

But, a warning: there's a potential problem with having the first item's responses recorded in the first column of the Data worksheet. What if someone doesn't answer an item? Some users let a blank, or empty, column represent missing responses. But a blank or empty first column in the Data worksheet has a very special meaning for Lertap: it indicates the end of data. Users who record item responses in column 1 of the Data worksheet should use a special code to cover the case of unanswered items -- for example, perhaps an "x", or maybe a "9".

For more about IDs, click here.

This revision also provides support for users who like to transform test scores, to re-scale them. Any formula may be applied to any score found in the Scores worksheet. Read all about it.


Apr 2004 (Ver. 5.4)

April 2004: consolidation (version change to 5.4)

We upped the version number to 5.4 for non-student users. This was done for two main reasons: we'd made a sufficient number of revisions to warrant a version number increase, and we installed a patch for an execution problem which previously dropped users into "Student mode", inserting an unwelcomed and unexpected new line in their Data worksheets in the process.


Feb 2004

February 2004: nothing sword-id about this:

Support for Bilog-MG users was added in September 2003. Now we've installed similar assistance for XCALIBRE users. XCALIBRE is an IRT program from ASC, makers of the well-known FastTEST item-archiving and test-generating system. (For more comments about FastTEST and Lertap, wiggle your mouse, and click here.)

About the same time, your favourite toolbar was enhanced. Yes. Part of it now sports Shorts. Check it out -- there's handy help for users who like to plot their output.


Nov 2003

November 2003: smiles for quintiles!

This is a BIGGIE, a considerable enhancement to Lertap's processing of cognitive test items. The upper-lower groups analysis module has been expanded so that as many as 5 groups may be processed. Two new types of charts are now available, plotting item results in some remarkably revealing ways. This revision is discussed under the Graphics trio topic.


Oct 2003 (Ver. 5.3)

October 2003: hope you like roots (version change to 5.3)

Added support for latent-root (eigenvalue) and SMC calculations.


Sep 2003

September 2003: to halve and hold forever (?)

An ability to create random samples of data records was added as an option under the Run menu. This will be useful, if not to the whole world, then to those who might use Lertap as a precursor to some subsequent analyses, such as, perhaps, IRT modelling. To ignore this revision, do not click here.

Another option added in September: support for those who love tetrachoric correlations, and Bilog-like data files.

And, late in the month the way The Spreader operates was changed.


Aug 2003 (Ver. 5.2.5)

August 2003: what you weighted for (version change to 5.25)

Two accreditation professionals, one in Puerto Rico, one in Florida, asked for an ability to credit an item even when it was not answered. It was possible to do this before, but now it's easier: the advanced toolbar works better, and a new form of the *mws card has been introduced, one which allows an "other=" weight to be applied.


Jul 2003

July 2003: changed item difficulty calculations

When a journal reviewer suggested alterations to Lertap's procedure for indexing the difficulty of a cognitive item, we allowed our arms to be twisted, and followed his advice. We made it possible for a cognitive item's difficulty to be computed in one of three ways. Read all about it with a click here.


Jun 2003

June 2003: added an item zapper

Sometimes there's a need to quickly remove an item from a subtest without having to re-do a bunch of CCs "cards". Previous versions of Lertap allowed this to happen, and we updated Version 5.2 so that it would, too. A wee click here will explain what we did.


May 2003 (Ver. 5.2)

May 2003: added "Lelp" (version change to 5.2)

Lelp is, of course, Lertap Help. It's what your peepers are feasting on at this very moment. When we took the leap and installed Lelp, we changed the version number to 5.2 (from just "5").


Resources

There are other Lertap resources.

There's the Lertap manual, which this document has made frequent reference to.

There's the Lertap website at Curtin University of Technology:

http://www.lertap.curtin.edu.au

The Lertap website has screeds of additional information, including sample data sets useful in measurement classes, or by people just launching their Lertap careers. The website also has a modest series of technical papers highlighting examples of Lertap applications, and discussing current developments (such as the experimental options in Lertap).

A history of Lertap is provided in the manual, and on the website. Lertap's pedigree goes back to the early 1970s. (You may not realise it, but you could have used Lertap in the past when it was masquerading under another title.)

And then there's always our support desk in sunny West Australia. We welcome questions and comments. Write to us at: support@lertap.com.


References

Please refer to the list of references found at the following URL:

http://www.lertap.curtin.edu.au/References.htm


Rchitect

aRchitect / authoR

Larry Nelson is not Lertap's only architect, but he's been the main one, having shepherded the system through several versions, and numerous host institutions.

Larry completed a BSc in electrical engineering at Wisconsin (1964); an MSc in Educational Psychology at Wisconsin (1970); and a PhD in Educational Psychology (psychometrics) at Colorado in 1973.

He's held a number of academic and non-academic posts in the United States, New Zealand, Venezuela, Thailand, and Australia. All have had something or other to do with applied statistics, data analysis, test development, data banking, and computers.

At November, 2006, Larry was thought to be wearing these hats:

Director, Lertap Project

Curtin University of Technology

Perth, Western Australia

Associate Professor (adjunct)

Faculty of Education, Language Studies, & Social Work

Curtin University of Technology

Perth, Western Australia

Professor (adjunct)

Department of Educational Research & Measurement

Faculty of Education

Burapha University

Bangsaen, Chonburi

Thailand

Driver

Miss Angela's Camping & Fishing Excursions

Jalbarragup, Western Australia

Lertap became an official project of Curtin University of Technology in 2001.

Click here for contact information.


About IDs

Lertap users often have ID (identification) information for each record in their Data worksheet.

Consider the example below, taken from a Data worksheet:

The first column of this sample Data worksheet (above) contains "Record" information, while the second column contains "ID" information. In this case, ID is a number. The ID for the first record in this Data worksheet is 9; the ID for the second record is 31.

Now look at another example:

The example above is taken from a data set developed by Dr Piet Abik, a chemistry teacher in Indonesia. In this example, the first column contains "nomor" (number), abbreviated as "No.". The second column is "ID Nama Siswa" (ID Student Name).

In both of these examples, the ID information found in the Data worksheet will be carried through to label the records in the Scores worksheet; for example, in Dr Abik's case, the top part of the Scores worksheet looked like this:

Not all Data worksheets have a column with ID information -- this information is not required by Lertap. However, when you want Lertap to use record IDs, it must be the case that the column's header begins with the letters ID, or id.

Now look at this example:

This in fact is how Piet Abik originally set up his Data worksheet. He wrote to us (at support@lertap.com) to ask why his Nama Siswa did not appear on the Scores worksheet; the reason is that the letters ID (or id) did not precede Nama Siswa.

Both of the examples seen here have a record number in the first column. This is not required -- the ID info can be in column 1. (In fact, as of July 2004, the ID info can reside in any column of the Data worksheet; yes, it can even appear after the item responses.)

Finally, users should remember that an empty cell in the first column of the Data worksheet is a special signal to Lertap: it means end of data. Note also that Excel will sometimes consider a cell with a numeric value of zero to be an "empty" cell.


defActiveWorksheet

Active worksheet

An Excel workbook may have many worksheets, but, at any given time, only one of them is said to be "active". An active worksheet is the sheet currently open for viewing. The best way to open a worksheet for viewing is to click on its tab.

In the screen shot shown below, the Stats2b worksheet is active:


defAffective

Affective item (definition).

 Return to Introduction

An affective item is a question which asks a person to report her or his attitude towards a particular subject, or, perhaps, her or his opinion of a certain state of affairs.

Affective item responses are frequently indicated on a Likert scale, from strongly disagree to strongly agree. Another popular format for affective items is the "semantic differential" one.

Affective items may use any response codes; a response code set of {1,2,3,4,5} is the default -- to specify other responses, an Res=() declaration is used on the *sub line in the CCs worksheet.

Affective items differ from cognitive items in that they do not have a single correct, or best, answer. On an affective item, Lertap awards points for each possible response to the item. For example, the "strongly agree" response might be scored as 5 points, while "strongly disagree" may be scored as just a single point. *mws lines may be used in the CCs worksheet to precisely control how Lertap scores affective items.

Unanswered affective items, often said to constitute "missing data" in this document, will generally be scored via Lertap's MDO substitution. To defeat MDO, the MDO control word is used on a *sub line in the CCs worksheet. More precise control over the scoring of unanswered affective items is achieved by using a *mws line with an other= assignment at the end.


DefASC

ASC = Assessment Systems Corporation (www.assess.com).


defASCII

ASCII = American Standard Codes for Information Interchange


defBilog

Bilog is one component in a series of IRT programs from SSI, Scientific Software International (www.ssicentral.com). In 2003, Bilog was replaced by Bilog-MG, an IRT program capable of processing multiple groups of test takers.


defCHM

chm = compiled help file, the native format of HTMLHelp systems as found under versions of Microsoft Windows.


defCognitive

Cognitive item (definition).

 Return to Introduction

A cognitive test item is one which has a correct answer, and is usually scored on a right / wrong basis.

Cognitive items may use any response codes; a response code set of {A,B,C,D} is the default -- to specify other responses, an Res=() declaration is used on the *sub line in the CCs worksheet.

The right answer to a cognitive item is usually scored as one point. To award more points, *wts or *mws lines are used in the CCs worksheet.

Unanswered cognitive items, often said to constitute "missing data" in this document, will be scored as zero points. To assign points to unanswered cognitive items, use a *mws line with an other= assignment at the end.

Cognitive items do not have to be scored on a simple right / wrong basis. To give points to more than one answer, use a *mws line in the CCs worksheet.

In some testing situations, cognitive items are "pre-scored". Please refer to the pre-scored items topic for further comments.


defEmuExport

Emu Export is one of the world's greatest lagers, brewed in West Australia.


defExcelHelp

Standard Excel toolbar

The toolbar which shows at the top of the Excel window has a number of options, including Help and Window. Help leads to the extensive online help resource built into the Excel system. Excel Help is usually the last menu option on the standard Excel toolbar, as seen below:

The amount of information available via Excel Help is vast.

The Window option will include a list of all Excel workbooks currently open.


defLelp

We use the name 'Lelp' for Lertap Help to distinguish it from Excel Help.


defManual

The Lertap manual

The Lertap manual exists in electronic and printed forms. An electronic version of the manual accompanies each copy of Lertap when downloaded from www.assess.com.

Extra electronic copies are available at no cost: download the whole manual by visiting www.assess.com (if you click here you'll see a picture of the downloads area within www.assess.com as found in December, 2006).

The printed version is available, at cost, from the same source, www.assess.com.

Some of the individual chapters in the manual may be freely downloaded from Australia: see www.lertap.curtin.edu.au.


defMinPos

MinPos and MaxPos

MinPos means minimum possible score. Every column in the Scores worksheet has a cell for MinPos. If the score in the column has been created by Elmillon, MinPos is the minimum possible score which anyone could obtain, as computed by Elmillon.

MaxPos, of course, refers to the maximum possible score which anyone could obtain.

The Scores worksheet also has a row headed "Min", and another headed "Max". Min is the lowest score obtained; Min can never be less than MinPos, but it can be, and often is, greater than MinPos. For example, on a test involving 10 addition problems, MinPos would be zero, but Min will be zero only if someone failed to get any of the problems correct.

Max refers to the highest score obtained on the test, or scale. The greatest possible value for Max is MaxPos. Max can never be greater than MaxPos; Max is often less than MaxPos, and, when this is true, it means no-one got a "perfect" score on the test or scale.


defMissingData

Missing data occurs when a person has not answered an item. In this case, a blank is usually entered in the Data worksheet -- however, users may enter any missing data code they want, as long as it is just a single character. For example, some people commonly enter a 9 (nine) to represent a missing response.


defMSLQ

MSLQ = the Motivated Strategies for Learning Questionnaire, University of Michigan, 1991.


defNathanCarr

The user was (Dr) Nathan T. Carr, of California State University, Fullerton. His review of Lertap in July, 2003, led to revisions in the way cognitive item difficulty is calculated.


defPDF

PDF = Portable Document Format, made popular by Adobe Systems.


defPreferredFonts

Lertap's preferred fonts are Verdana and Courier New. Almost all of Lertap's reports look best with Verdana. The Arial font is generally a good second choice if you don't have, or don't like, Verdana.

(We chose Verdana as it's said to be a good common font, one which displays well on both Windows and Mac computers.)


defPrintedManual

The printed manual is on A4 paper, and has 203 pages. It's from the Faculty of Education, Curtin University of Technology, Perth, Australia, ©2000. It may be ordered from ASC in the United States (visit www.assess.com).


defRecord

The term "record" means a row in the Data worksheet. There is said to be a "record" for each person who responded to the test or survey; in other words, there is one row of Data information for each person.


defMaxiumResponseWeight

Lertap uses the term "weight" in numerous spots. Response weights, for example, are simply the number of points given when a person selects the corresponding response. Thus, "maximum possible weight" refers to the most points a person can get for her/his answer to an item.


defRightClick

Right-click means to click using the right mouse button (not the left one, which is the usual one).


defSAQ

SAQ = seldom-asked question


defSPSS

SPSS refers to the Statistical Package for the Social Sciences. Please refer to www.spss.com for more information.


DefSubtest

Subtest definition

A "subtest" is a group of items. Each *col line in the CCs worksheet defines an item group. If the items are cognitive ones, the group is usually referred to as a "subtest". If the items are affective ones, the group of items may again be referred to as a "subtest", or as a "scale", or as a "subscale", depending on the user's preference.

Lertap will produce a score for each subtest or scale. Such scores are found in the Scores worksheet produced when the "Elmillon item analysis" is taken from the Run menu.

Some users often ignore Lertap's subtests, scales, and scores. These users are usually those who want to look mostly at item response frequencies -- for example, how many people got Item 12 right, or how many answered strongly disagree at Item 32. For such users, Lertap produces too much information -- for them, the simple report given in the Freqs worksheet is at times sufficient. At other times, the data in a Stats1b worksheet will do, with the interpretation of results aided, perhaps, by item response charts.

For additional reading on subtests, scales, and the interpretation of Lertap results, please refer to Chapters 7 and 8 of the manual.


defSubWorksheet

Sub worksheets are produced when the "Interpret CCs lines" option is taken. They're usually hidden from view as their contents are crudely formatted, and, generally, only useful to the Elmillon routine. To see them, go up to the Excel toolbar, then click on Format / Sheet / Unhide.


DefSystemWorksheet

The System Worksheet

The System worksheet is one of the worksheets found in the Lertap5.xls workbook -- you can see its tab down towards the bottom of this screen snapshop:

The Comments worksheet is displayed above. To see the System worksheet, just click on its tab.

The System worksheet looked like this as of June, 2004:

We used to ship Lertap with the System worksheet hidden in order to discourage less experienced users from fooling around and getting into trouble. Trouble is especially likely to result if the settings in Rows 3, 4, and 5 are altered.

If one of the Present Settings is changed, most often the Lertap5.xls workbook has to be closed and then re-opened before the change will be effected. However, there are some settings which take effect immediately, and which do not require closing Lertap. Among these are the settings seen in Rows 12 through 15, and in Rows 17 through 21 above.


defThen

"Then" refers to the years 1972 to 1974. A Ford Foundation project encouraged the development of technical resources for use in Venezuelan public education. Elmillon was one of several Spanish-based data analysis routines produced by the project.


defUnhide

Unhiding worksheets

To unhide a Lertap worksheet, follow these steps:

First, try using Excel's Format / Sheet / Unhide. If that doesn't work, try the following:

Use Excel's Tools menu / Protection / Unprotect Workbook.

The unprotect code is shack25

Then, use Excel's Format / Sheet / Unhide.

Lertap commonly hides at least two worksheets, Sub1 and System.


defURL

URL = uniform resource locator (typically an address to an Internet resource of some sort, such as an "http://" address).


noteNewMdo

Revised default missing-data weight for affective items.

 Return to Introduction

The computation of the default missing-data weight for affective items, as described in the manual, has been updated. At 20 August 2003 the default missing-data item score (or weight) for affective items was changed so that it now is equal to the mean of the item response weights.


SetiadiProblem

Cannot find Sub worksheets message.

 Return to Introduction

At times Lertap may report that it "cannot find any Sub worksheets to run with".

This message usually results after the "Elmillon item analysis" option has been selected from Lertap's Run menu.

Sub worksheets are normally created when the "Interpret CCs lines" option is taken. However, under some circumstances they will not be made.

There are times when users will have a CCs worksheet with only one card, or line: *col. This is done when users simply want to see Lertap's Freqs report to check on response frequencies. In this case, most users know that the "Elmillon item analysis" option will not work, and may realise that Sub worksheets have not been generated.

Sub worksheets may not be made if Lertap has encountered an error in the CCs lines. One error known to lead to this situation is that wherein users have omitted the "Aff" control word from the *sub card, as seen in the following example:

 *col (c28-c37)

 *sub title=(Comfort)

 *pol +---- ++--+

In this little example, the user wanted to process affective items, but neglected to include the Aff control word. The correct CCs lines are shown below:

 *col (c28-c37)

 *sub aff, title=(Comfort)

 *pol +---- ++--+

Should you encounter this message, or a similar one, never hesitate to write for support: send an email message to support@lertap.com and we'll get back to you as soon as we can.


Start Spreading?

The Spreader has worked down the cells in its first pass, checking string lengths. It will have told you if it found any string whose length seemed to be short.

Now it's ready to start the second pass down the cells; on this pass each string will be taken apart, with its characters distributed, one by one, over the cells to the right.

Take note of where The Spreader will start from on its second pass, okay? If you stopped The Spreader at least once during the first pass, the start point may be further down the rows than you might expect.

Click the No button if you don't want The Spreader to make a second pass.


ToolbarIconDelete

Deleting secondary worksheets

ToolbarIconDelete

This icon deletes certain worksheets from a Lertap 5 workbook.

To understand what it does, and why it is used, consider a standard Lertap 5 job: a user puts his/her item responses, and perhaps other data, into a workbook's Data worksheet. S/he puts control statements, or control "cards", into the CCs worksheet.

The user then clicks on the Run drop-down menu, and selects the "Interpret CCs lines" option. What happens? Lertap looks at the rows in the CCs worksheet to find out which columns in the Data worksheet the user wants to analyse, and then it goes on to read data, and add new worksheets to the workbook.

The most obvious worksheet added by this step is the one called "Freqs".

The user pauses to scan the information found in the Freqs worksheet. If all is in order, the user usually returns to the Run menu, and clicks on "Elmillon item analysis".

What happens? Lertap adds more new worksheets to the workbook, such as "Scores", "Stats1f", "Stats1b", and so on.

These worksheets which Lertap adds are called secondary worksheets. The original Data and CCs worksheets are referred to as primary worksheets. At any time, the secondary worksheets may be regenerated if the primary worksheets are still on hand.

What the little ToolbarIconDelete does is allow the user to quickly delete the secondary worksheets. When this icon is clicked on, a dialog box appears which informs users that "all worksheets whose names begin with freqs, scores, sub, stats, and histo will be deleted" if the user clicks on the OK button.

Why would a user want to delete the secondary worksheets? Probably to save disk space. The secondary worksheets can be quite large if there are many items in the data set, and/or if there are many records in the data set. Users may also want to delete the secondary worksheets if they want to send a copy of their workbook to someone else.

Ordinarily, no harm is done by using ToolbarIconDelete. Going through the process of "Interpret CCs lines" and "Elmillon item analysis" will restore the secondary worksheets at any time. Users who wish to make sure their secondary worksheets will not be deleted by this icon should rename the worksheets, perhaps by putting a prefix before their names. For example, Freqs might be renamed as OrigFreqs. Note that renaming Freqs as FreqsOrig would do no good as the worksheet's name still begins with Freqs -- a prefix is suggested, not a suffix.

How to rename a worksheet? The fastest way is to right-click on the worksheet's tab at the bottom of the screen. To read about the various ways which may be used to rename worksheets, please refer to Excel Help.


ToolbarYellowSmiley

The yellow smiley face.

 Return to Introduction

ToolbarIconSmileyFace

Use this icon to get the Lertap 5 version type, and date of generation.

The screen snapshot showing above indicates that the "ASC" version of Lertap 5 was in use. The version number is 5.25, created on the 8th of August, 2003. (This snapshot was taken when Lertap 5 was running with Excel 2002, working within Microsoft's "XP" operating system. The snapshot will look a bit different when Lertap is running on other versions of Excel.)

Lertap 5 version data may also be obtained by clicking on "About this version", an option which displays when you click on the LertapHelp menu drop-down from Lertap's toolbar.