Assessing PLE/LMS systems

We are starting the processing of evaluating our commitment to our current LMS. We have done some informal surveys in the past but would like to do a real needs assessment to determine what system would best serve our needs - be it Blackboard, Moodle or something else. 

Has anyone gone through this process recently and how did you go about gathering input - surveys, focus groups, something else? Any advice or examples would be a tremendous help. As an aside, what did your school end up going with? 

Jason Mellen (Bowling Green State University)

Annelie Rugg (UCLA)
UCLA undertook a major assessment of LMS needs for both instruction and collaboration, leading to the choice of Moodle as the underlying software for our Common Collaboration and Learning Environment (CCLE). The entire process is documented at http://oit.ucla.edu/ccle/. Some of the documents in the lefthand navigation menu are not publicly viewable, but it should give you some ideas on what we were looking for.

Melanie Kroening (Mesa Community College)
We are getting ready to start the process so I am gathering similar data as well. Here are some additional links I'm finding useful. The Mohawk one I find to be the one I like best as a model as it involves the faculty and students more, I know I the past we haven't interviewed other schools using the systems so I am interested in ensuring we do a vendor 'background check' of sorts this time around as well as its interesting to hear from similar schools what they think and I think that's a piece we've missed in the past.

http://lmsreview.mohawkcollege.ca/rfp.html (in progress)
http://www.csuchico.edu/tlp/LMS2/minutes.shtml (2005)
http://www.ltrc.mcmaster.ca/lmseval/index.html (April 2008)
http://information.anu.edu.au/daisy/infoservices/1804.html (in progress)
http://at.ufl.edu/~cmsag/ (2002)
http://www.isu.edu/itrc/resources/moodle-info.shtml (2007)

FYI right now we run Blackboard CE 6.2.3. In the past we've done a simpler review process, we started with a general survey of faculty, have had open demos by the vendors with faculty being invited, and then had follow-up surveys and would forward all data to the administrators for decision making. We are talking about faculty focus groups at the beginning of the process this time around so I'd be curious how you structure your review as well :-)

Donna Petherbridge (North Carolina State University)
NC State University's process for moving from WebCT Campus Edition to WebCT (now Blackboard) Vista is documented at:
http://delta2.ncsu.edu/slic/

This move was not "recent" (e.g. 2003/2004), but there are some documents there they may be of use to you.

Currently (2008), we have a Moodle Pilot Implementation Team, comprised of representation among the Distance Education & Learning Technology Applications (DELTA) group, and 4 college representatives - who are instructional/academic technology staff. To begin looking at Moodle, we held two focus groups in October 2007, and invited as many groups as we could think of, getting the "go-ahead" from our faculty & partners that yes, looking into open source was a good idea. You can see the documentation for this project at: http://wikis.lib.ncsu.edu/index.php/LMS_Strategy and note that we've had recent open update sessions as well as several assessments that gather pilot user input , and plans to put together committee structures to help us continue to get user input, no matter what our final decisions are related to what LMS we use.

PS I am at Educause and went to Louisiana State University's session today on their selection of & transition to Moodle. See their info at http://appl027.lsu.edu/itsweb/cmsweb.nsf/

Otto Khera (USC)
See the 2006 MIT Report: "MIT Peer Comparison on Course /Learning Management Systems, Course Materials Life Cycle, and Related Costs."
http://web.mit.edu/emcc/www/MIT-WCET-C-LMS-Final-Report-07-19-06.pdf

We found this to be very helpful here at USC.

DI von Briesen (Central Piedmont Community College)
As the resident Moodle bigot, I can tell you that the following schools on my radar have switched to moodle (just a sampling, but I’m sure they did some analysis prior to the jump).
 

• Guilford Technical Community College (NC) (Amy Brown is contact)
• Isothermal Community College (NC)
• Blue Ridge Community College (NC) (Alice Crisp is contact)
• Louisiana State University
• Lane Community College (OR)
• Athabasca U in Canada
• Open University in the UK

 
And many others… Then there are some (like my school) that operate Moodle alongside Blackboard- adoption seems to be almost entirely based on which instructional developer/faculty member new employees work with first. There is an advocacy forum on moodle.org at http://moodle.org/mod/forum/view.php?id=2784 (you may need to create a login to access this, but you should anyway if you’re doing this research)
 
What’s interesting to me is that each school ends up doing their own evaluation, when in fact most schools have the same needs and the same choices, and at the end of the day, any matrix you come up with will not really capture the differences. For example, you can check off that the tools have forums, but what about number of steps to grade each forum? Ability to organized threads? Ability to link to student profiles from each thread, and so on.
 
You will of course also find people become attached to what they know, for better or for worse. This tunnel vision makes it hard to look with fresh eyes at a different product. I personally feel that the only way for an individual to know is to teach (not evaluate, or check off, but TEACH) classes with each platform. Administrators will base their decision on other things, like support, ease of modification, etc…
 
In terms of collecting data, I’d highly recommend google forms, which make it very easy to create, publish and collect results from users with no fee or limit of responses- and as part of google docs spreadsheets you can do some basic charting analysis.
 
An interesting approach might be to look at schools that did NOT switch after doing the evaluation, and figure out why. My experience has been that the biggest nut seems to be conversions of existing courses. There exist no easy and straightforward tools to do entire conversions – though we do them with a combination of tools – but even those take some heavy tweaking.
 
Finally, be sure to factor in elements of community with your analysis- i.e. to what extent can one join user groups, get help or discuss teaching with other people with regards to that platform.

Milt Nielsenn (Texas State University)
Two, almost three years ago, we did a review on different systems, looking at replacing Blackboard for a lack of growth in tool set, less than the best support and rapidly increasing cost.  Consequently, those issues were at the top of our list of characteristics to be supported by a new LMS.  As we opened our list to open source, while exploring the cost issue, we  also included concerns about long term viability, cost if any existing support structure we would rely upon suddenly disappeared, the cost of “going it alone,” the probability of “going it alone,” design that supported continuous addition of viable tool sets introduced by faculty and students, and scalability to deal with expected growth in student population and use of an LMS.  Database flexibility also showed up.  At that point Moodle had not developed to its current state.  We selected Sakai.  Moodle is really liked for its simplicity of use.  In contrast, Sakai, is liked for its customizability and complexity in that it allows for broadened and more variance in its use as a student or faculty project communication tool amongst other things.  Our list seemed at that time to best fit with Sakai.  We have not started our formative system evaluation at this point, so I cannot comment on the growth of  Moodle, which based upon comments of some of our contemporaries is fairly significant.  I would just emphasize doing as detailed front end analysis to include many areas of growth and all the really costs so you don’t have to replace again soon, especially if you choose an open source tool and make the associated commitments.  
 
A major source of input was the faculty requests and complaints about the current system.  Inputs from our instructional technologies advisory group (which includes our early innovators as well as some technologically challenged) were also helpful.  We also looked at the marketing of commercial LMS’s and evaluated their “new developments” for viability and value.  Subsequently, we had our help desk team and the advisory group evaluated our list of expected characteristics, as did as our instructional design and software production teams. 
 

Kathy Fernande (California State University)
From CSU Chico, we have put up our rubrics and our committee notes on how we decided 3 years ago to go with our "Next Generation Learning Management System."
http://www/tlp/LMS2/  While the specifics of the rubric and evaluation questions could change, the process was very good.
 
Meanwhile, at the CSU Chancellor's Office, we did both an RFI and RFP (Request for Proposal) process early this year (2008). This process is not about the Chancellor's Office deciding what the 23 CSU campuses will use for an LMS. Rather, this process is to qualify the vendors and products to ensure we've done our due diligence and surveyed the market appropriately. Due to our strict accessibility issues in the CSU, only Moodle and Angel made the cut the first time. We are currently in another round of doing an RFP for the CSU. This process was very helpful to the CSU campuses to have all this information available to consider while determining what process their campus would use to make a decision on their future LMS. Not every campus has done their strategic planning on this but every campus who has used a different process to determine their outcome based on campus technology committees and processes already established at their campus to make academic technology decisions.

[Attached] is a copy of our current CSU LMS Request for Proposals. This document doesn't "give answers" but does cause the institution to reflect and question all aspects of an LMS so that appropriate consideration can be given to all aspects of choosing an LMS for the future.