Chapter 9. Training, Feedback and Issues Arising

9.1 Introduction

In this section, we first provide some details on the interviewer training provided for this specialised survey relating to disability. We then document the manner in which the questionnaires were completed and how long they took. We then focus on feedback and reactions from both the respondents and interviewers to the survey. Finally, we point to a few areas where the precise format of questions employed in the pilot questionnaires might be amended, and to the overall implications of the experience of the pilot in terms of feasibility.

9.2 Interviewer Training

At the design and planning stages of the pilot survey it was felt that interviewers used on the survey would need some specialised training in dealing with persons with a disability, both to ensure the interaction was appropriately sensitive and to maximise the quality of the response.

In advance of the survey all interviewers were given a half-day briefing session by Survey Unit staff from the ESRI. This training session covered 3 main areas as follows:

  • Background and context;
  • Sensitivity training in interviewing persons with a disability;
  • Protocols and procedures for implementing the instruments;
  • Substantive issues arising in the instruments.

The first section of the training session included a detailed discussion of the background and general context of the survey as well as general issues arising. This involved providing the interviewer with standard background or contextual information in respect of the sponsor of the survey (NDA); the rationale for the survey; the origin of the sample (derived through the sifting process at the first stage of the project).

The second section of the training session involved sensitivity training. The relevant instruction material for this session included a detailed section on 'Good Practice in Interviewing Persons with Disabilities'. This (given in full in Appendix 5) dealt with the following issues:

  • Definition of disability;
  • Informed consent;
  • Meeting people with disabilities - general points;
  • Meeting people with hearing disabilities;
  • Interviewing people with hearing disabilities;
  • Meeting people with physical disabilities;
  • Interviewing people with physical disabilities;
  • Meeting people who are blind or partially sighted;
  • Interviewing people who are blind or partially sighted;
  • Meeting people with speech and language difficulties;
  • Interviewing people with speech and language difficulties;
  • Meeting people with Specific Learning Disabilities;
  • Interviewing people with Specific Learning Disabilities;
  • Meeting people with an intellectual disability (general learning disability);
  • Interviewing people with an intellectual disability (general learning disability);
  • Meeting people with an emotional or mental health disabilities;
  • Interviewing people with an emotional or mental health disabilities;
  • Meeting people with hidden disabilities;
  • Interviewing people with hidden disabilities.

The third part of the training session addressed detailed operational issues on survey implementation as well as the substantive content of the instruments used. This latter involved going through each of the two instruments used in the main survey as well as the sifting questionnaire used to capture the target sample. Each question was discussed and explained (where necessary) to interviewers. Time was spent in attempting to convey to all interviewers the exact nature of the information being sought by and recorded in each question. For quality control purposes it is essential to ensure that all interviewers working on a survey have a full understanding of the issues involved and the detail of each question.

When interviewers began fieldwork their first two completed questionnaires were returned for special examination by data coding and checking staff. Detailed written comments were sent to interviewers on any problems which are found to have arisen in their first interviews. All interviewers were alerted to potentially systematic problems if/when they were identified. All interviewers were issued with photo ID cards and a specially prepared brochure outlining details on the survey. These were given to the respondent immediately on initial contact, helping to establish the interviewer's bone fides and seriousness of purpose.

The use of alternative methods of communication for respondents affected by certain impairments, and the appropriate training for interviewers in these alternative methods, may also arise in a survey of this type. In the pilot exercise interviewers were briefed in detail about such situations, but due to the small size of the pilot sample and the limited numbers in the population affected by such conditions the need for, for example, assistive technology or signing did not arise in practice. However, in a full-scale national survey these special considerations at the operational level would clearly need to be taken into account.

9.3 Questionnaire Completion

Adult Questionnaire Completion

A total of 88 of the 93 interviews with adults were completed directly by the respondent. One interview was a facilitated interview (the respondent was present but was assisted in completing the interview by a parent/guardian/facilitator) and four interviews were completed by proxy.

As shown in Table 29, the average length of the adult questionnaire was just under 40 minutes, with a median of 35 minutes and a range from 15 minutes (2 cases) to 80 minutes.

Table 29a: Completion of Adult Interviews
MethodNumber%
In person8895%
By Proxy44%
Facilitator11%
Total93100%
Table 29b: Length of Adult Interviews
MeasureMinutes
Mean39
Median35
Minimum20
Maximum80

Child Questionnaire Completion

Only 1 of the 13 child questionnaires completed was based on direct responses from the child, the remainder being conducted with the parent or guardian of the child. Table 30 shows that the average and median length taken for completion were both 41 minutes, with a range from 25 to 65 minutes.

Table 30a: Completion of Child Interviews
MethodNumber
In person1
By Proxy4
Facilitator8
Total13
Table 30b: Length of Child Interviews
MeasureMinutes
Mean41
Median40
Minimum25
Maximum65

9.4 Respondent Feedback

In order to learn more from the pilot exercise, the respondents’ reactions to the survey were investigated in a structured way on both the adult and child questionnaires. Each respondent was requested to complete a feedback section, and was given the choice of filling this out with the interviewer immediately after the interview itself took place or on a confidential basis on a separate sheet left with the respondent for completion and return to the ESRI in a pre-paid envelope. A total of 90 out of the 106 respondents completed the feedback section of the questionnaire.

Satisfaction with Interview

The figures in Table 31 generally indicate high levels of satisfaction with the interview process in terms of courtesy of the interviewer, clarity and pace of the instrument and also relevance of the content. One can see, however, that 6 per cent (representing 5 of the 90 relevant respondents) did express some level of dissatisfaction with the relevance of the questionnaire.

Table 31: Respondent Feedback
 

Very Satisfied

Satisfied

Neither Satisfied nor Dissatisfied

Dissatisfied

Very Dissatisfied

Courtesy86%14%0%0%0%
Clarity80%19%1%0%0%
Pace77%23%0%0%0%
Relevance51%36%7%6%0%

In addition to the above, information was also recorded on views relating to length of the interview; issues not covered and dissatisfaction with specific questions. These latter 2 items were recorded on a verbatim basis.

Length of Interview:

Only out of 90 respondents felt it was ‘a little long’. All other respondents thought it was about right.

Issues not Covered

Some extremely useful comments were received here. Over one-third of respondents mentioned at least one issue that was relevant to their situation but that was not covered. There is no clear pattern – each issue tended to have been mentioned by only one respondent. In view of the length of the questionnaire, perhaps the potential issues (medication, costs of medication/equipment to respondent, availability of information etc.) could form the basis of further consultation prior to fielding the survey nationally.

Discomfort with Questions

In general, the sensitivity of the questions appears not to have been a problem. Of the 88 respondents who answered this item, only 3 said that they were uncomfortable with particular questions. One mentioned age, two mentioned the questions on financial situation, and one noted that depression/mental illness was a sensitive topic generally.

9.5 Interviewer Feedback

As well as respondents, we felt that it was important that interviewers also fill out a feedback form for each interview. Interviewers provided feedback in respect of 99 out of the 106 interviews completed. As can be seen in Table 32, high levels of satisfaction were found with regard to layout, clarity, detail and sectional balance.

Table 32: Satisfaction with Interview
 

Very Satisfied

Satisfied

Neither Satisfied nor Dissatisfied

Dissatisfied

Very Dissatisfied

Layout36%56%8%0%0%
Clarity33%59%6%2%0%
Detail28%60%9%3%0%
Sectional Balance26%61%11%1%0%

Length of Interview:

Out of the 99 completed interviews, 5 interviews were felt by interviewers to be ‘too long’ with the remaining 94 being ‘about right’.

Issues not Covered

As with the respondent feedback no clear pattern emerged, with different items being mentioned by different interviewers. Items suggested for inclusion ranged from medical (questions on progression of the health condition) to more questions geared towards social participation, i.e. attitudes of society. Table 33 shows that interviewers rated the level of interest of the respondent and also the clarity of the questions positively (92 to 93 per cent). Between 7 to 8 per cent gave indifferent ratings.

Table 33: Level of interest and clarity for Respondent
 

Very Positive

Positive

Neither Positive Nor Negative

Total

Level of interest60%33%7%100
Clarity of Questions53%39%8%100

9.6 Implications

Having implemented the survey with the training and supports outlined above, interviewers then reported experienced no particular problems in the field arising from the specific nature of the survey. No particular problems for those who responded were raised in the respondent reaction section to the survey. In addition, an ex post 'spot check' on approximately 15 per cent of respondents was carried out. The purpose of this is to ensure that the survey was conducted in a satisfactory and professional manner for the respondent. No problems were identified in these spot checks. On this basis, our overall assessment is that a survey of this type can be feasibly conducted by fieldworkers drawn from a panel of interviewers trained and experienced in work on general surveys oriented towards the social sciences, given appropriate training.

Finally, some minor issues relating to the precise wording of specific questions on the pilot questionnaire may be mentioned. One of the issues highlighted in the consultation process described in Section 5 (rather than in the survey respondent feedback) was the need to avoid any suggestion that a medical model of disability underpinned the exercise. Moving away from that model is of course central to the thrust of the ICF itself, as we have discussed, and to the way the pilot questionnaire’s structure was framed. However, a few of the questions employed might inadvertently be open to misinterpretation as reflecting a medical model perspective, which could be avoided by rewording.

This applies specifically to four questions in the pilot questionnaire. The first, on education, read (see Appendix 3, question P12):

“Did you discontinue your formal education or training because of your condition ?”

One clearly has to be particularly careful about the causality implied, and about the role of barriers facing people with disabilities. A better way to phrase this question might be:

“Was your education or training adversely affected by because of issues arising from your condition?”

The second, on the main reason for leaving a previous job (question Q1 part 4, adult questionnaire), gives a list of potential responses including “own illness or disability”; this might perhaps be reworded to:

“Peoples' reactions or barriers raised in relation to your disability”

The third asked (Question Q2 part 4) :

“Because of your condition have you ever changed the amount of work you do, … the kind of work you do … ?”.

This might be better worded as:

“Were you forced, as a result of peoples' reactions or barriers raised in relation to your condition, to change the amount of work you do... the kind of work you do …?”

Finally, the pilot (Question Q2 part 5) asked

“Do you believe that your condition makes/made it difficult for you to change jobs or to advance at your job?”

This might perhaps be better as;

“Do you believe that peoples' reactions or barriers raised in relation to your condition make/made it difficult for you to change jobs or to advance at your job?”

Having made these minor amendments, we believe the questionnaire as piloted provides a sound base for a full-scale national disability prevalence and impact study. We go on in the next section to discuss how the results from such a questionnaire could be mapped onto the framework provided by the ICF.