About Betatesting


Overview

A shell is the user's interface with the system, and directly influences his or her experience with it. It needs to be easy to use, stable, and professional appearing. Finding and fixing bugs is not enough - scrutinizing SharpE for improvement in any of those three areas is a constant process.

You must be on the lookout for three things while testing:

  • Identifying bugs (actual errors)
  • Improving usability (is it intuitive?)
  • Creating a professional experience (does it look like a hackjob?)

To start, pick one aspect of the shell to test at a time. It's easier to evaluate a single module or "user experience" than to collect data from the entire shell all at once. In addition to using the shell, set aside an hour or two to examine parts of it thoroughly. Today, see what you can do with the scmd module; tomorrow, explore the hidden aspects of SharpBars?, or play with the menu. If you try and do too much at once, you'll get tired and miss things; if you don't take it seriously enough, you'll build your own subconscious workarounds. Make your reports clear and concise, and include screenshots where they would be helpful.

Remember, everything can be improved - nothing is perfect or final.

Bugtesting

Bugs can destroy a project's reputation faster than anything else. When evaluating for bugs, do whatever you can to break the part of the shell in question. Do things you don't expect the average user to do. Load 150 plugins at once; close a critical SharpE process in the middle of using it. If a config asks you to fill in some value, put in something you know is wrong. And while you do this, make sure you note the steps you are taking.

If you find something, restart the process and try to do it again. If you can reproduce it, great! Make a bug report and clearly identify all the steps you took to produce that error. Note details such as what build you are using, what version of Windows, and anything running on your system that might have affected it.

If you can't reproduce the error, make certain you state that in the report.

Usability & Consistency

Usability is difficult to test once you've used a project for awhile. You become competent at doing things a particular way, although there may be a much better way of doing it. Every time you attempt a task (creating a theme, editing a SharpScript, moving a plugin on the bar), ask yourself: Is there a more intuitive way I could do this? Better yet, grab a sibling or parent, a friend who doesn't use SharpE, and see if they can figure it out. Some things will always require practice, but every task should be as easy and intuitive as possible.

Also, actions should be consistent within the shell. If a right-click opens a context menu on one plugin, it should open the menu on all plugins. Make sure that each plugin or action conforms to a shell standard.

Sometimes developers ignore these, falling into the same trap described above - "We've been doing it this way for x number of releases, I've never had any complaints, and it works fine." Even though rewriting a fully functional part of the shell is a lot of work, the gains in usability can be phenomenal, and can be the difference between a new user staying with the shell or switching to a competitor.

When reporting usability issues, describe where you saw the issue and your vision for its improvement, in as much detail as possible. Even if not accepted for inclusion, the insight might cause the developer to rethink a particular feature.

Examples:

  • How many clicks does this take?
  • Would I have to refer to documentation to accomplish this as a new user?
  • Is the layout of this dialog easy to read, or do I have to scan several times to find the options I want?
  • Is this behavior consistent with what I expect on a Windows system, or is it alien and unfamiliar?

Professionalism & Appearance

The last thing may crop up less frequently, but is the final stop for spit-and-polish of the shell. How are sentences worded on a dialog? Are they clear, concise, spelled correctly and use appropriate grammar? On Yes/No dialogs, are the buttons in the same order, and is the same button the default on each one? Not paying attention to these details make the project look like a rag-tag assortment of random parts assembled by amateurs from all over the world. Whether this is true or not is irrelevant - SharpE needs to present a consistently professional appearance. If it looks like it was designed by sloppy coders, people will assume it was designed by sloppy coders, regardless of what's "under the hood."

When submitting a report about this, make sure you identify clearly which dialog/screen you noticed it on, and the steps taken to open that screen; then, suggest an alternative.

Example Reports

Bug Report
Version: 0.7.2
System: Windows XP Professional SP2

Issue: In the SearchBar? module, hitting the "Enter" key while the search field is blank generates the exception: (paste exact error message contents)

Suggestion: There should be no action in this scenario; if the search field is blank and the user presses "Enter", it should ignore that keystroke.

Usability Report
Version: 0.7.1
System: Windows 2000 SP4

Issue: Dragging a plugin on the SharpBar? provides no visual feedback to the user.

Suggestion: Change the default mouse cursor to a hand that closes on mousedown ("grabbing" the throbber) and opens on release and hover. This provides the user an experience consistent with Windows behavior.

Professionalism Report
Version: 0.7.3
System: Windows Vista

Issue: The confirmation dialog on overwriting desk settings states: "Are you sure you want to save over the settings?"

Suggestion: "Are you sure you want to override your current settings? You cannot undo this operation."