Evaluation and survey methods
An evaluation in its strictest sense is a comparison between the goal that was set and the result that has been achieved. For cultural institutions however, the notion of evaluation is much greater: it is both a method and the overall term for several forms of surveys and investigations. Audience research or visitor studies, though specific kinds of evaluation, are often understood as interchangeable terms.
The empirical social sciences developed a number of methods that may be used for evaluation and audience research. They differ mainly according to the nature of the data gathered and the character of the targeted groups.
Quantitative and qualitative data
Quantitative methods are most commonly used for evaluation, especially in surveys. The aim is to measure and quantify the frequency of occurrence of a particular variable in a specific sample. This implies a sufficient number of data sets so as to enable a solid statistical analysis. Only on this basis can statements be formulated as objectively as may be. Instruments of quantitative studies are for instance (partly) standardised questionnaires – be they written surveys, interviews or online surveys – or observation protocols.
Qualitative data requires a smaller number of data sets. The information gathered is not necessarily representative of the whole target group or of general validity. Subjective points of view and individual statements can provide extremely valuable information as well, one suggestion might be the beginning of a successful undertaking. Instruments of qualitative research are, for example, interviews with or without guideline, expert panels, focus groups or observations of a small sample.
The target groups
Defining the target group and the sample of the survey is an important methodical decision. The validity of the whole survey depends on it. The criteria for inclusion and exclusion must be carefully considered in accordance with the goals of the evaluation. Those criteria guarantee that the sample is on the one hand free of any structural bias – as far as possible – and on the other hand that it is representative of the whole target group. In other words: only those, who are really able to make a statement, should be asked for it.
Depending on the content of the survey, the target groups might be visitors or non-visitors, all the users of a cultural institution or just some of them. To make the choice, the defined criteria might be for example age (children, teenagers, adults and/or senior citizens), company (individual visitors or organised groups), specific characteristics (members, subscribers, occasional users, tourists or local population, experts, etc.) or specific behaviour (visitors to certain parts of the exhibition, only on weekdays or on weekends, using a hands-on exhibit, etc.). A further question of method is whether the respondents have been informed of the survey ex ante or ex post.
The sample size is another characteristic feature of the evaluation. It is both dependent on the methodical requirements of a survey – quantitative data needs much larger samples than qualitative data – and on the dimension of the target group. Provided the latter has just the required scale for the sample, then all individuals will be asked to take part to the survey. Sampling will be necessary in case of larger target groups.
Two different methods are currently used for sampling. The first consists of defining quotas based on some relevant characteristics previously known, such as the composition of the institution’s audience for instance, or important for the survey – as an example: a sample consisting exactly evenly of men and women. The second method consists in a random selection of respondents: a specific rule is defined beforehand, such as asking every third or every tenth visitor to answer the questionnaire. In this case, the size of the sample may be reduced in comparison to the target group without any loss in its being representative or without the introduction of any bias.
Go back to benefits