Friday, 29 August 2008

A conceptual model of e-learning: better studying effectiveness

My personal development has recently led me to explore and research the effectiveness of e-learning approaches to Higher Education (HE) teaching and learning. Since the late 1990s, e-learning has become a key focus of activity within pedagogical communities of practice (as well as those within information systems and LIS communities who often manage the necessary technology). HE is increasingly harnessing e-learning approaches to provide flexible course delivery models capable of meeting the needs of part-time study and lifelong learners. Of particular relevance, of course, is the Web, a mechanism highly conducive to disseminating knowledge and delivering a plethora of interactive learning activities (hence the role of informaticians).

The advantages of e-learning are frequently purported in the literature and are generally manifest in the Web itself. Such benefits include the ability to engage students in non-linear information access and synthesis; the availability of learning environments from any location and at any time; the ability for students to influence the level and pace of engagement with the learning process; and, increased opportunities for deploying disparate learning strategies, such as group discussion and problem-based or collaborative learning, as well as delivering interactive learning materials or learning objects. Various administrative and managerial benefits are also cited, such as cost savings over traditional methods and the relative ease with which teaching materials or courses can be revised.

Although flexible course delivery remains a principal motivating factor, the use of e-learning is largely predicated upon the assumption that it can facilitate improvements in student learning and can therefore be more effective than conventional techniques. This assumption is largely supported by theoretical arguments and underpins the large amounts of government and institutional investment in e-learning (e.g. JISC e-learning); yet, it is an assumption that is not entirely supported by the academic literature, containing as it does a growing body of indifferent evidence...

In 1983, Richard E. Clark from the University of Southern California conducted a series of meta-analyses investigating the influence of media on learning. His research found little evidence of any educational benefits and concluded that media were no more effective in teaching and learning than traditional teaching techniques. Said Clark:
"[E]lectronic media have revolutionised industry and we have understandable hopes that they would also benefit instruction".
Clark's paper was/is seminal and remains a common citation in those papers reporting indifferent e-learning effectiveness findings.

Is the same true of e-learning? Is there a similar assumption fuelling the gargantuan levels of e-learning investment? I feel safe in stating that such an assumption is endemic - and I can confirm this having worked briefly on a recent e-learning project. And I am in no way casting aspersions on my colleagues during this time, as I too held the very same assumption!!!

It is clear that evidence supporting the effectiveness of e-learning in HE teaching and learning remains unconvincing (e.g. Bernard et al.; Frederickson et al.). A number of comparative studies have arrived at indifferent conclusions and support the view that e-learning is at least as effective as traditional teaching methods, but not more effective (e.g. Abraham; Dutton et al.; Johnson et al.; Leung; Piccoli et al.). However, some of these studies exemplify a lack of methodological rigour (e.g. group self-selection) and many fail to control for some of the most basic variables hypothesised to influence effectiveness (e.g. social interaction, learner control, etc.). By contrast, those studies which have been more holistic in their methodological design have found e-learning to be more effective (e.g. Liu et al.; Hui et al.). These positive results could be attributed to the fact that e-learning, as an area of study, is maturing; bringing with it an improved understanding of the variables influencing e-learning effectiveness. Perhaps electronic media will "revolutionise" instruction after all?

Although such positive research tends to employ greater control over variables, such work fails to control for all the factors considered – both empirically and theoretically - to influence whether e-learning will be effective or not. Frederickson et al. have suggested that the theoretical understanding of e-learning has been exhausted and call for a greater emphasis on empirical research; yet it is precisely because a lack of theoretical understanding exists that invalid empirical studies have been designed. It is evident that the variables influencing e-learning effectiveness are multifarious and few researchers impose adequate controls or factor any of them into research designs. Such variables include: level of learner control; social interactivity; learning styles; e-learning system design; properties of learning objects used; system or interface usability; ICT and information literacy skills; and, the manner or degree to which information is managed within the e-learning environment itself (e.g. Information Architecture). From this perspective it can be concluded that no valid e-learning effectiveness research has ever been undertaken since no study has yet attempted to control for them all.

Motivated by this confusing scenario, and informed by the literature, it is possible to propose a rudimentary conceptual model of e-learning effectiveness (see diagram above) which I intend to develop and write up formally in the literature. The model attempts to improve our theoretical understanding of e-learning effectiveness and should aid researchers in comprehending the relevant variables and the manner in which they interact. It is anticipated that such a model will assist researchers in developing future evaluative studies which are both robust and holistic in design. It can therefore be hypothesised that using the model in evaluative studies will yield more positive e-learning effectiveness results.

Apologies this was such a lengthy posting, but does anyone have any thoughts on this or fancy working it up with me?

Friday, 8 August 2008

Where to next for social metadata? User Labor Markup Language?

The noughties will go down in history as a great decade for metadata, largely as a result of XML and RDF. Here are some highlights so far, off the top of my head: MARCXML, MODS, METS, MPEG-21 DIDL, IEEE LOM, FRBR, PREMIS. Even Dublin Core – an initiative born in the mid-1990s – has taken off in the noughties owing to its extensibility, variety of serialisations, growing number of application profiles and implementation contexts. Add to this other structured data, such as Semantic Web specifications (some of which are optimised for expressing indexing languages) like SKOS, OWL, FOAF, other RDF applications, and microformats. These are probably just a perplexing bunch of acronyms and jargon for most folk; but that's no reason to stop additions to the metadata acronym hall of fame quite yet...!

Spurred by social networking, so-called 'social metadata' has been emerging as key area of metadata development in recent years. For some, developments such as collaborative tagging are considered social metadata. To my mind – and those of others – social metadata is something altogether more structured, enabling interoperability, reuse and intelligence. Semantic Web specifications such as FOAF provide an excellent example of social metadata; a means of describing and graphing social networks, inferring and describing relationships between like-minded people, establishing trust networks, facilitating DataPortability, and so forth. However, social metadata is increasingly becoming concerned with modelling users' online social interactions in a number of ways (e.g. APML).

A recently launched specification which grabbed my attention is the User Labor Markup Language (ULML). ULML is described as an "open protocol for sharing the value of user's labor across the web" and embodies the notion that making such labour metric data more readily accessible and transparent is necessary to underpin the fragile business models of social networking services and applications. According to the ULML specification:
"User labor is the work that people put in to create, improve, and maintain their existence in social web. In more detail, user labor is the sum of all activities such as:
  • generating assets (e.g. user profiles, images, videos, blog posts),
  • creating metadata (e.g. tagging, voting, commenting etc.),
  • attracting traffic (e.g. incoming views, comments, favourites),
  • socializing with other people (e.g. number of friends, social influence)
in a social web service".
In essence then, ULML simply provides a means of modelling and sharing users' online social activities. ULML is structured much like RSS, with three major document elements (action, reaction and network). Check out the simple Flickr example below (referenced from the spec.). An XML editor screen dump is also included for good measure:

<?xml version="1.0" encoding="UTF-8"?>
<ulml version="0.1">
<channel>
<title>Flickr / arikan</title>
<link>http://fickr.com/photos/arikan</link>
<description>arikan's photos on Flickr.</description>
<pubDate>Thu, 06 Feb 2008 20:55:01 GMT</pubDate>
<user>arikan</user>
<memberSince>Thu, 01 Jun 2005 20:00:01 GMT</memberSince>
<record>
<actions>
<item name="photo" type="upload">852</item>
<item name="group" type="create">4</item>
<item name="photo" type="tag">1256</item>
<item name="photo" type="comment">200</item>
<item name="photo" type="favorite">32</item>
<item name="photo" type="flag">3</item>
<item name="group" type="join">12</item>
</actions>
<reactions>
<item name="photo" type="view">26984</item>
<item name="photo" type="comment">96</item>
<item name="photo" type="favorite">25</item>
</reactions>
<network>
<item name="connection">125</item>
<item name="density">0.167</item>
<item name="betweenness">0.102</item>
<item name="closeness">0.600</item>
</network>
<pubDate>Thu, 06 Feb 2008 20:55:01 GMT</pubDate>
</record>
</channel>
</ulml>


The tag properties within the action and reaction elements are all pretty self-explanatory. Within the network element "connection" denotes the number of friend connections, "density" denotes the number of connections divided by the total number of all possible connections, "closeness" denotes the average distance of that user to all their friends, and "betweenness" the "probability that the [user] lies on the shortest path between any two other persons".

Although the specification is couched in a lot of labour theory jargon, ULML is quite a funky idea and is a relatively simple thing to implement at an application level. With the relevant privacy safeguards in place, service providers could make ULML files publicly available, thus better enabling them and other providers to understand users' behaviour via a series of common metrics. This, in turn, could facilitate improved systems design and personalisation since previous user expectations could be interpreted through ULML analysis. Authors of the specification also suggest that a ULML document constitutes an online curriculum vitae of users' social web experience. It provides a synopsis of user activity and work experience. It is, in essence, evidence of how they perform within social web contexts. Say the authors:
"...a ULML document is a tool for users to communicate with the web services upfront and to negotiate on how they will be rewarded in return for their labour within the service".
This latter concept is significant since – as we have discussed before – such Web 2.0 services rely on social activity (i.e. labour) to make their services useful in the first place; but such activity is ultimately necessary to make them economically viable.

Clearly, if implemented, ULML would be automatically generated metadata. It therefore doesn't really relate to the positive metadata developments documented here before, or the dark art itself; however, it is a further recognition that with structured data there lies deductive and inferential power.