Friday, 12 April 2013

To-do list, as of 12 April 2013

  1. Flesh out HK preso for pre-confirmation: 'A few days prior to the meeting of the panel, students should provide the supervisor and other member of the panel with a written document briefly outlining their work and progress in the last year (including planned Research Questions, data, data collection and data analysis) and pointing out any issues which they consider have impeded their progress (3-4 pages).' 
  2. Finish reading Understanding digital literacies: A practical introduction.
  3. Sort out literature (spreadsheet, Zotero) -- what's the best way?
  4. Find at least 2 schools -- follow up on possible candidates.
  5. Design draft questionnaire for students (for MOE application).
  6. Work on ethics and permissions docs.

Misc stuff to follow up, as of 12 April 2013

From CR forum exchange with D:
Could the conventional concept of a construct could also be challenged if it assumes a theoretical perspective of 'practice' rather than 'skill'?

Case studies with ethnographic perspective? According to U on DL forum: 'Call it taking an ethnographic perspective - in the sense that you seek the emic perspective, look at things in context/as situated, so you share something with ethnography, but doing it in a more condensed and probably less 'deep' way. Lots of people these days speak about their research being ethnographic, but that doesn't mean it is ethnography in the sense anthropologists define it. Often what they do is qualitative research or case study research.'

Or multi-sited ethnography? U: 'The issue is that the research questions might lead you into different directions, requiring you to look in different places, hence multi-sited. These directions might not be physical or virtual places, but rather issues or questions - hence more abstract.'

HK presentation (Feb 2013)

On the last lesson of the Classroom Language Assessment module, I presented my latest research plan to the class. L says it can be developed into research plan for pre-upgrade in July.


Hot topics in educational research but not so much in language assessment
Issues are LINKED
Issues central to lifelong learning (self-assessment, SDL, learner autonomy) and to preparing us for this age of exponential changes

Increasing importance of ICT in teaching & learning
Singapore MOE's 3rd Masterplan for ICT in Education:
'...enrich and transform the learning environments of our students and equip them with the critical competencies and dispositions to succeed in a knowledge economy.'
'Strengthening integration of ICT into Curriculum, Assessment and Pedagogy'

Future Schools testbeds for innovation

New syllabus acknowledges multimodal literacies, e.g. Writing and Representing, not just writing.
Doc mentions 'visual elements', 'multimodal texts', 'experiment with the use of sound, imagery and language', 'use a range of technologies'.

Importance of 21st C skills, e.g. various digital literacies (c.f. critical competencies & dispositions)
Singapore is a founding member of ATC21S (Assessment & Teaching of 21st C Skills)

How it's being assessed and WHAT is being assessed (i.e. the construct) important

Literacies & assessment as social situated practices - must be considered along other social & cultural practices
Literacies (multiple) not as skills but as activities
Skills view does not capture the diversity and complexity of people’s reading and writing activities in everyday life
Think about all the kinds of writing you do everyday, esp tech-mediated

Assessment practices also not neutral, but also shape and are shaped by society & culture

"A ‘practice’ (Praktik) is a routinized type of behaviour which consists of several elements, interconnected to one other: forms of bodily activities, forms of mental activities, ‘things’ and their use, a background know-ledge in the form of understanding, know-how, states of emotion and motivational knowledge." Reckwitz (2002)

Notes from Feb 2013 HK meetings

New RQs foregrounding e-assessment:

RQ1. What role does ICT play in the assessment practices of EL teachers in Singapore secondary schools?
RQ2. How do they balance formative and summative e-assessment in the classroom?
RQ3. What are the factors that enable and inhibit their use of formative e-assessment?

To be considered:
  • Refocus on digital writing instead (because e-assessment difficult to rationalise for research).
  • Rationalise digital writing research from literacy practices perspective, 21st c edtech development in last 5 years
  • Literacies and assessment as social practices, not skills
  • Challenges to construct of writing.
  • Instead of including formative/summative in RQ, refer to them simply as practices.
  • Grounded Theory -- Lincoln and Guba (1984)?
  • Theory of Planned Behaviour TPB -- Azjen (social psychologist)
  • Innovation Theory as an orientating theory (?) to start with
  • Paul Gruba on e-assessment and blended learning

Notes from Jan 2013 Residential

My RQs at this date:

RQ1. What are the assessment practices of EL teachers in Singapore secondary schools actively engaged in integrating ICT in teaching and learning?

RQ2. How do they balance formative and summative assessment in the classroom?

RQ3. What are the factors that enable and inhabit their use of formative assessment?

RQ4. What role does ICT play in their (formative) assessment practices?

New focus on e-assessment (mostly writing) requires definition:

  • MOE?
  • Research literature?
  • What about e-portfolios?

Consider infrastructure and resources as factors.

Consider literature on curriculum reform, particularly those studies which are concerned with situational or environmental analysis (essentially, those studies that look at factors which enable and inhibit). Also innovation theory in ELT (see Markee's book).

Notes from July 2012 Residential

July 2012 Residential

Meeting 1

Explore the ethnographic approach in language assessment (e.g. Hill & McNamara, 2012)
Consider what sort of participant schools would suit
Read up on 
  • Case studies (see Yin)
  • Situated Literacies ed. Barton
  • Assessment literacy
  • Teachers' understanding of formative assessment
  • Teacher cognition (Borg, 2003)
  • Definitions of formative assessment

Meeting 2

Fulcher's focus on PRINCIPLES is important and would apply to informal assessment
Should decide what to foreground and background in study
Could use formative vs summative CBA as a lens through which to examine factors which inhibit or enable formative assessment (e.g. learner autonomy, exam-driven culture, assessment literacy)
Follow up on:
  • Classroom-based Language Assessment 
  • Kathryn Hill's research
  • 2013 Language Testing special issue on Assessment Literacy

Meeting 3

Tentative RQs:
RQ1. What are the assessment practices of EL teachers in Singapore?
RQ2. How do they balance formative and summative assessment in the classroom?
RQ3. What are the factors that enable and inhibit their use of formative assessment?
Should not bring factors I'm interested into the RQs (e.g. washback), but choose one as a focus RQ, e.g. assessment literacy, because it can make a considerable contribution to PD?

Framework for CBA: TIERNEY 2006Changing practices: influences in classroom assessment
Model missing resources and materials (ready made)
Note mediating sources - variables

Follow up:
  • Black and Wiliam on formative assessment
  • Van lier
  • Focus on formative - precise end point (narrow to writing? Digital writing?) in order to include all influences
  • Use model to analyse data?
  • Ethnographic methods for classroom research

My original proposal

B. Your Research

B1. What is your proposed area of research?
Language testing. Specifically the testing of non-traditional digital writing skills, as defined by researchers in the field of multiliteracies (New London Group, 1996). The context is secondary school English language education in Singapore.

B2. Within this area, what would you like to investigate? Please include at least one possible research question here.
  1. How can a given construct of non-traditional digital writing skills be operationalised in terms of assessment criteria and rating scales? 
  2. In the operational test, to what extent do the individual criteria distinguish between test-takers who score different grades? 
  3. Are raters experienced in the assessment of traditional writing skills able to use the new criteria consistently and reliably? 
  4. Does the use of such a test result in desired washback; that is, improvements in the instruction of non-traditional writing skills? 

B3. What are your reasons for choosing this proposed research project? (These might be theoretical, professional or both.)
The nature of literacy has changed with the advent of affordable computing devices and higher internet speeds. These new forms of literacy are fast becoming essential in modern society. However, mainstream education continues to lag behind research in this area. This is despite the strong push by education authorities in many countries, such as Singapore, to integrate ICT into their literacy programmes.

In Singapore’s assessment-driven education system, this problem could be attributed to the traditional assessment practices still dominant here. Crucially, the high-stakes Singapore-Cambridge General Certificate of Education (Ordinary Level) Examination for English Language, taken by secondary school students in their final year, remain very much a traditional language test. Therefore, assessment practices would have to take the lead in changing the status quo.

However, research in the assessment of multiliteracies has been limited to date. My research aims to fill the gap in a way that is immediately applicable in the context of Singapore schools. This is in line with Singapore’s role as one of the five founding countries of the Assessment and Teaching of 21st Century Skills project (ATC21S).

B4. Do you have ideas about the sort of data you might need for your research?
  1. Test-takers’ scripts from a new test of non-traditional digital writing. 
  2. Double-rated scripts by teacher-raters. 
  3. Think-aloud protocol (concurrent or retrospective) data from teacher-raters. 

B5. How might you collect this data? Do you already have access to it? (N.B. Do not list here books you will read or have read.)
The participants will be teachers and pupils from a secondary school in Singapore who are also participating in a study that aims to identify similarities between the skills involved in traditional pen-and-paper writing and non-traditional digital writing.

Why this blog

For a long time now I've considered putting down thoughts, etc. related to my PhD studies in some way, in a kind of journal. As usual I've procrastinated and procrastinated.

Let's hope I can keep this up!