Project Name: Mobile checklists enabling authentic workplace assessment


Case Study

A digital story (below) has been developed by the project team to give you a quick summary of how, what where this project was developed.










‍Public Contact Details


Pauline_SMALL.jpg
Name: Pauline Farrell
Role in the project: Project Director
Email: p.farrell@boxhill.edu.au
Phone: 9286 9237

Other team members:


Julianne_Small.jpg
Name: Julianne Seaman
Role in the project: Project Manager
Email: j.seaman@boxhill.edu.au
Phone: 9286 9344

Name: Andrew King
Role in the project: LMS Coordinator
Email: a.king@boxhill.edu.au

Project Profile:


Who are the learners and how will they benefit from this project?



Air Conditioning and Mechanical Contractors' Association (AMCA) and Box Hill Institute (BHI) are committed to broadening delivery locations and finding simple ways to better integrate the workplace. This project allows us to shift from traditional classroom based delivery to workplace based delivery in a National priority training area. Apprentices are most suited for this project where work undertaken through industry placements provides the opportunity for learners to demonstrate their level of skills, knowledge and experiences in a contextualised workplace environment.

By utilising a blended approach, learners from carpentry, plumbing, automotive, and drafting will have opportunities to develop their employability skills and readiness for work in an environment congruent to their preferred delivery mode and learning styles. BHI apprentices are highly visual and kinaesthetic and up to 30% of our trades students have some degree of learning disability. By increasing visual and kinaesthetic learning opportunities we can better retain and engage at risk learners or those with significant learning differences.

In 2012, students in construction have been provided iPad’s. This has provided capacity for BHI to extend our level of engagement with employers and learners by integrating work-based mobile assessment. By providing access across a range of mobile devices using PhoneGap, and offline and online data storage capabilities, students, employers and teachers can facilitate remote and location based assessment and data collation utilising mobile checklists. The expansion of the satellite NBN provides scope for making this project feasible with photos, video, and audio for learners in remote workplace locations.

Where are your learners located?

Box Hill Institute students undertake study in Melbourne, Victoria, whereas AMCA students are located across
Australia. Each of these groups undertake learning through a combination of:
  • Classroom based delivery at Box Hill Institute
  • On location at the AMCA head office
  • Workplace based learning
  • Mobile or computer based online learning

This project will require BHI trainers to visit the worksites in local or remote locations to verify and authenticate the assessment tools being submitted. Employers or workplace supervisors at these locations will also fulfil a necessary component of this project, thereby promoting better communication between the teachers and employers.

How will training be delivered?

A multimodal delivery approach will be adopted using face-to-face classroom, workshop, and workplace based delivery supported by the use of mobile devices to capture and record video, photos, audio and observation checklists into the Moodle Learning Management System.

Describe the importance of this project for your target learners and explain how it will meet the strategic objective of the E-learning for Participation and Skills funding to utilise innovative e-learning practices and / or high speed broadband-enabled technologies to deliver e-learning projects and strategies which improve individuals’ access to gain pre-vocational skills, skills in priority employment areas, or to pursue educational pathways to gain higher qualifications?


Box Hill Institute is focussing on transforming trades from the traditional approach to training and delivery to a model that better integrates workplace learning, encourages input from industry, and adopts mobile learning and assessment practices congruent with our learners needs and preferences. To extend the capacity for authentic workplace learning and assessment, we would like to investigate ways mobile devices can be used to record and transfer workplace assessments (both offline and online) into the Moodle LMS for targeted apprenticeship areas. This involves the development of digital observation checklists, offline accessibility for remote access, and synchronisation with Moodle when an Internet connection becomes available using PhoneGap. Furthermore, research into ways teachers, employers, and students can attach videos, photos and audio to checklists from mobile devices will be undertaken. By developing digital checklists that integrate a range of media the feasibility of utilising e-learning in the workplace to record student progress is strengthened.

To transfer or synchronise digital observation checklists containing video, photos, and audio with Moodle, a high speed wireless or fixed broadband connection is recommended. Learners in remote workplace locations will benefit from accessing broadband satellite over the NBN.

What are the intended outcomes from your project, and how you will measure them?


The project outcomes will be:
Investigation into:
  • building a mobile application with PhoneGap, a widely interoperable mobile browser
  • storing assessments including video, audio, and photos locally on mobile devices and synchronising information with the Moodle
  • authenticating and tracking PhoneGap data
  • Design and development of twelve formative assessment tools that include:Development of HTML, JavaScript and CSS templates for PhoneGap
  • observation checklists
  • third party reports
  • question and answer checklists
  • Trial with thirty students
  • Design and delivery of a training program for teachers and employers
  • Documentation in integrating PhoneGap with Moodle
  • Monthly reporting of project activities and outcomes in reference group and staff meetings.
  • Evaluation through statistical analysis of feedback from students, teachers and employers, and Moodle tracking.

Stakeholder outcomes:
  • Learners: Increased employability and digital literacy skills through opportunities to undertake workplace learning and assessment; closer alignment to students preferred modes of delivery and learning styles; authentic learning experiences promoting higher levels of student engagement.

  • Employers: greater level of engagement with the RTO and better capacity to negotiate customisations of the program to meet their learners needs

  • RTO: Build capacity of knowledge around mobile learning, PhoneGap and synchronisation with Moodle from which to extend upon; Opportunities to compel external and internal stakeholders to adopt flexible workplace based models; greater engagement with employers; higher levels of student retention.

In conjunction with meeting the e-standards, a primary focus will be to meet the e-standards for
mobile platforms as outlined at: http://e-standards.flexiblelearning.net.au/technical_standards/mobile_platforms.php

What are the needs of the target learner group/s and relevant stakeholders (employers, community, training organisations etc)?


AFLF National Indicators (2011) have identified that 80% of BHI students want more e-learning and blended
learning. Internal analysis has shown that:
  • 98% of our building, electrical and automotive students have a smartphone
  • 90% of our apprentices have access to wireless internet at home
  • 75% of our apprentices have a lap-top
  • building students have the lowest proportion of linguistic learners.

This has significant implications as the current resources used by our apprenticeship students are highly text-based. Internal analysis have found that BHI apprentices are predominately kinaesthetic, social and visual learners, a demographic that is well suited to mobile assessment in workplace environments. This delivery and assessment mode can enable, or evidence, capture of authentic learning and aid in fast tracking their apprenticeship. In working with AMCA Box Hill Institute has acknowledged similarities in learning preferences of Certificate IV in drafting students and AMCA has indicated their high level of interest in participating in this program, identifying the benefits for their learners, and particularly those in remote locations.

Describe the learning approach, program and technologies to be used... (Include details about where the training and trials will take place, the number of learners and the qualification, units(s) or skills sets to be delivered.)


Formative assessment tools including observation checklists, third party reports and question and answer checklists will allow employers to capture evidence of informal learning that can be formalised through the design of better mobile delivery and on-site assessment. The project will utilise PhoneGap to capture assessment information (including video, audio, and photos) and the Moodle LMS to collect and store mobile data. Phone Gap, a framework that allows hybrid applications to be built using web technologies (HTML5, CSS and JavaScript) will allow BHI to build a mobile application that is interoperable across 7 different types of mobile devices including Apple, Android, Windows Phone 7 and Blackberry. Once installed, the application can access devicefeatures including its camera, GeoLocation and Accelerometer, enabling video and photo collection. Where no internet connection is available, PhoneGap can utilise local application storage space. Programming will be required to:
  • allow data, including locally stored logs, to automatically upload and synchronise with Moodle
  • implement user authentication
  • 200 learners are expected to access this program, and it will be trialled with 30 students across the following areas:
    • Cabinet Making: MSAPMOHS200A Work safely
    • Automotive: AURC270103A Apply safe work practices
    • Electrical: UEENEEE101A Apply OHS regulations, codes and practices in the workplace
    • Refrigeration: VU20874 Maintain essential safety measures
      * Note: Following a request from our client, this unit was used instead of the unit
      MEM09201A Work effectively in an engineering drafting workplace.

BHI and AMCA subject matter experts will be integral in the redesign of existing assessment tools so that they integrate work-based formative assessments including observation checklists, third party reports, and question and answer checklists that can be accessed by trainers and workplace assessors.

Please list 5 key words (metadata) that people can use to find your project:

mobile, checklists, workplace, apprentices, assessment



Monthly Project Updates
September 2012
Project team formed and project planning completed.
Wikispaces site has been updated.
October 2012
During October our infrastructure team commenced research into suitable ways that we could achieve the project aims from both Apple and Android devices.
In conjunction to this, we commenced internal marketing by presenting the project to Teaching Centre Managers and the M & E-Learning Working party. A positive reaction resulted with real benefits being realised from a learner, teacher and industry perspective for our apprenticeship areas. The feasibility of this project extending to other levels of study across a range of courses was also discussed as a potential future objective.
November 2012
Across November and December our primary focus was to continue undertaking research and commence designing the assessment tools.
The basis of our initial research was to investigate PhoneGap and competing systems including Titanium and Rhodes to ensure that it would be the most appropriate app development framework for our project.
Based on our project objectives, the following infrastructure considerations needed to be met:
  • Means to develop an app that is interoperable across a minimum of 2 main mobile application platforms, preferably iOS and Android which caters for a large proportion of SmartPhone users.
  • One code base with dual publishing for rapid publishing
  • M-standards and e-standards compliance
  • Ability to synchronise with Moodle LMS system
  • Ability to create assessment templates that area editable by teachers
  • Ability to utilise smartphone native functionality such as the camera and offline, local storage.
In conjunction to this, the key criteria that we were looking for in terms of app platform solutions would be one that could leverage existing skills, offer the most sustainability in terms of being able to reuse the same code, and the ability to extend to other mobile devices if needed.As a result of the research it was determined that PhoneGap provided the best mix of these criteria.
The immediate advantages we had found in choosing PhoneGap are that it:
  • supports HTML5/CSS3/JavaScript
  • can integrate existing JavaScript libraries/frameworks such as Jquery
  • write once, deploy everywhere methodology
  • has been used for the development of existing moodle apps
  • is an open source system
  • has a common API set to access phone functionality such as camera
  • supports eight platforms including iOS, Android, Windows Phone and BlackBerry
The disadvantages that were found include the following:
  • It has a limited native API
  • The interface is reliant on the quality of the web renderer for the platform
  • There are cross browser limitations just like regular desktop web development.
December 2012
Learning design was required to design the assessment templates, align assessment tasks to each template, and create a user process.
The templates that were deemed most useful by our stakeholders and most achievable by the project team were:
  1. Job Safety Analysis
  2. Digital story
  3. Knowledge test, including multiple answer questions
  4. Observation checklist
Corresponding assessment tasks were developed, resulting in 12 formative assessment tasks across the following units:
  • Cabinet Making: MSAPMOHS200A Work safely
  • Automotive: AURC270103A Apply safe work practices
  • Electrical: UEENEEE101A Apply OHS regulations, codes and practices in the workplace
  • Drafting: MEM09201A Work effectively in an engineering drafting workplace. (AMCA Unit)
The user experience was determined through consultation with the whole project team to ensure that there was a consistent user flow that reflected M-Standards, Moodle logic and navigation flow, WCAG2.0 guidelines in terms of accessible design and navigation considerations.
Based on prior knowledge of the Access to Skills project ‘AS091 E-profiling Smart Phone App’, the user experience was documented in interactive HTML prototypes using the mobile wireframe software Balsamiq Mockups. This proved to be a highly effective way to establish the user process and test early in the piece whether there was a logical and intuitive user progression through the assessments.
January 2013
Mobas.jpg
Mobas Home
Multimedia design commenced to establish a look and feel for the application. Many staff were on leave across January which meant that resources for this month weren't readily accessible for further development.

After further discussions with AMCA, the Drafting unit was replaced with an OH&S unit for refrigeration students to VU20874 Maintain essential safety measure. Our learning designer worked with subject matter experts to update the assessment tasks and templates.
February 2013
We are now under way in terms of app development from both the application and Moodle side. There has been a big learning curve for the team in terms of web services, flexibility of PhoneGap, access to core mobile functions, and effective ways of using local storage for media including photos, video and audio.
Due to the size of video files we have not found a suitable way to locally compress and store this information across a range of devices. We are focussing on gathering photo and text based data at this stage and will then investigate options for video and audio later in the project.
There has also been concerns around the degree to which we can align our project to the E-standards, however we are currently working through our concerns with the ELP&S, primarily in relation to the use of HTML5 and applicability of the standards to mobile app development.
March 2013
We saw it as imperative that any other stakeholder considerations that may have been missed were identified before the final build of the application was done.
To gather feedback on project design, alignment to industry, learner and teacher needs, we met with our Industry partner, and the Centre managers of the apprenticeship areas. Overall the feedback was positive in that the application was seen as an opportunity to provide greater opportunities for learners to engage in what would typically be paper based assessments in the workplace through their mobile phones, minimising the associated paperwork and extending the opportunities for workplace based assessment. AMCA was particularly keen to promote this app as a national standard for assessment in the refrigeration industry.
Across March we also spent much time in forming arrangements for learner trials with learners across our four course areas, and this also involved asking learners to complete associated parental consent forms and their pre-trial survey.
April 2013
The development team encountered many challenges across April in regards to using Web Services to transfer information in and out of Moodle, managing local caching on the mobile devices, and also in regards to how best to create a plugin for the assignment module that wouldn't affect Moode's core code. The overall strategy was to come up with a solution that was simple to use, had the capacity to expand, and would be sustainable. The time required to uncover and then overcome these challenges was significant and well beyond the time that was anticipated for the application build. As a result the trials that had been booked for this month needed to be set back to another month. Fortunately however our apprenticeship groups come into the institute through rolling enrolments so this was possible to set the trial date back another month.
May 2013
The project prototype was presented at the Moodle-Mahara Meetup in Adelaide on the 7th of May. Positive feedback was received around the usability, relevance to VET, and linkage to Moodle. During or after the session, candidates could interact with a web version of the application allowing them to trial responding to questions and attaching photos to question responses.

The build process was still encountering many hurdles. The team had successfully created to assessment template types: JSA and Digital Story. The knowledge test proved to create many challenges however as it was potentially requiring our developers to replicate Moodle quiz functionality and embed this within an assignment, or alternatively use the Moodle quiz functionality which would require other work for it to appear in the Gradebook in the same way as the other assessment types. The observation checklist also presented the challenge of needing a significant amount of time to create a teacher view that could populate and authenticate observation checklists from the teacher/supervisors/assessors device and send this to Moodle. Since time was running short, the team needed to revisit the assessment templates that were being developed and determine if any templates could be developed that extend on the coding that had already been done. Through a discussion with the project team, it was agreed that the following changes would take place:
  • Digital story - The existing code would be used to create 2 different activities instead of the Digital story - A Procedure/process and Work Diary.
  • Knowledge test, including multiple answer questions - This template would be removed.
  • Observation checklist - This template would be modified so that it could be completed as a 'Demonstration checklist' on the learners device. The issue of authenticity could be overcome through the provision of an 'Assessor code' that needs to be entered into the students device prior to submission of the assessment activity.

As a result, our learning designer and multimedia designer was involved again and redesigned all applicable assessment tasks to instead align to one of the following templates that would be handed over as our deliverable:

  1. Job Safety Analysis
  2. Procedure/Process
  3. Work diary
  4. Demonstration checklist

Learner trials were conducted across the areas of Automotive and Refrigeration using a web based version of the app, however from a user perspective the web based version is perceived to be fully contained as an application and is intended to fully mimic the Apple or Android versions. From the trials we received feedback from learners that the application was intuitive, usable, and something learners would like to use as part of their assessment. Learners could see themselves using this in the workplace and viewed that a big win was the ability to easily transfer photos into the student system, which has been a frustration in the past.Teachers were highly impressed by the learner uptake, however the big win from a teaching perspective was the ability for them to view submissions, grade, and provide feedback to learners all through the Moodle LMS. Furthermore, teachers learnt within 30 minutes how to create/modify/update assessment tasks using the Mobas assessment templates aiding in the level of buy in we received for future uptake.

Areas that needed to be improved included the amount of instructional text provided on the mobile interface and how this should best be provided.

The application was presented for a second time at the National VET eLearning Strategy project showcase event on the 30th of May. The Moodle and application sides were presented and again positive feedback was received around it's applicability to many assessment contexts.
June 2013
Still split amongst many competing Institute priorities, our infrastructure team was working hard to complete the build of the Mobile assessment application.

Once all assessment types were built, the next stage was to package the application for Android and iOS devices. Due to the time constraints we were only able to create an Android package, however have also made available a web version that retains the look and feel of using a smart phone/tablet application interface.

The Mobas app and Moodle plugin will be provided to the strategy but all the files will also be placed in GitHub, a cloud based repository where files that are in a module can be modified and changes can be downloaded to system administrators as they are available.

The GitHub URL's that can be used to access these files are:

The application was presented for a third time at MoodleMoot13 in the context of changing assessment practice and improving the student experience.
Project wrap up
We would like to thank the National Vet eLearning Strategy for the opportunity to build our internal capability in the area of mobile application design and development, resulting in the delivery on an application and web app that communicates with Moodle. Out of this project we developed the following four templates, which can be customised to align to any assessment context by teachers in Moodle:

MOBAS2.gif

Recent discussions with centre managers and teaching staff from our three key trades areas indicated that they are positive to embed mobile assessment using Mobas into their training and assessment plans and have expressed interest in setting up training for teachers in targeted areas. Outside of the trades areas, our senior educators have all been introduced to the application and as a result, our Animal Sciences teachers have expressed interest in the customisation and integration of assessment tasks for their distance and flexible learners.

This project has allowed Box Hill Institute to report the following key successes:
  • Robust discussion at the executive level at how this tool can be used to drive pedagogical changes particularly in the Trades areas to radically improve the percentage of workplace based assessment
  • Teaching teachers to design more flexible and innovative assessment tools by stealth.
  • Promoting the use of assessment tools that live within Moodle to assist the Institute in the area of audit
  • Promoting the use of mobile assessment tools based on students feedback so we better meet their needs
  • Promote greater communication and transparency between the employers and institute
  • Promote greater communication and transparency between the employers and their apprentices/ trainees
  • Provide a point of differences with both our Industry partners and potential new or existing clients
  • Upskill current staff so they had both design and development skills in application development in this emerging area, opening the possibilities for future app development at BHI.
  • Upskill the whole staff into understanding that mobile assessment and delivery is a core business we move into this emerging era created an understanding that projects of this magnitude and complexity needed all team members to equally contribute to achieve the outcomes required
  • This project delivered better working relationships across teams that often didn’t need to work closely together and built a trust and respect that will service us well into the future for other more complex projects.
  • Further extend our relationship with AMCA and the refrigeration industry
  • From a design perspective, increased capability to design storyboards for mobile devices using Balsamiq mockups for wireframe to prototype and aligning the navigation structure to Moodle LMS.
  • The positivity by teachers around the simplicity of the re-usable templates purporting that uptake will follow.
  • Extended opportunities workplace assessment and remote learners