Appropriate handling of collected data
Careful, anonymous, reliable, trustworthy and confidential: the collected data requires honest and conscientious handling. Such are the ethical rules of statistics, such are the expectations of any institution I work with and such are my own requirements. Those rules are prerequisites for the quality of the evaluation. I commit myself to conforming to them and to setting them as standards for the handling of the collected data.
Surveys are strictly anonymous
Questionnaires are always completely anonymous. The processing and analysing of the collected data are strictly anonymous as well. Name, contact details such as postal address, telephone number or email address, membership number or identification code of any kind, none of these ever appear in the list of questions to be answered in the survey.
There are nevertheless some person-related data collected for the sake of evaluation. Who could describe a sample without at least some basic information about the individuals it consists of? Gender, age, education level or place of residence (country and town, usually through its postcode) for instance are fundamental factors often accounting for certain behaviours, needs or expectations. Though personal, those data are always collected in such a way as to exclude any allocation to an identifiable person. The statistical processing of data makes personal identification impossible as well.
The qualitative elements of the survey are also processed anonymously. Some respondents may include person-related information in their answers to the open questions of the questionnaire, if any. Some respondents might also spontaneously add unasked for comments or inscriptions in the margins of the questionnaire. None of this information will be processed so as to identify a real person, most of this information will not even be taken into account for data processing and analysis.
The survey remains strictly anonymous even in the case of a namely known target group. Should the questionnaire be sent out by electronic or by postal mail for instance, the list of contact details remains with its owner and will not be passed on to the researcher: the mailing is managed by the cultural institution with a strict partition with the processing of the data done by the evaluator.
Qualitative surveys: careful handling of personal data
Qualitative surveys usually imply a more personal contact with the respondents. Depending on the goals and methods of the evaluation, very specific target groups or selected individuals are to be approached. It then becomes necessary to rely on some pieces of personal information, in various cases even to give up the respondents’ anonymity entirely. Identifying the respondents personally is crucial for focus groups, expert interviews or panel-surveys for instance. The selection criteria, and sometimes the contact details, belong to the collected and processed data.
The sources of those personal data are many. Getting them from the cultural institution itself is no exception. As an alternative, it may also be part of the evaluator’s mandate to find out the appropriate (potential) respondents. In all cases, those are data that have to be handled with particular care – including as part of the legitimate protection of personal data set by the law.
Personal data is only communicated to the cultural institution in case of a previous agreement with the evaluator and provided the persons concerned have given their consent to it. In no case those personal data will be communicated to third parties not involved in the evaluation.
„Glaube keiner Statistik, die Du nicht selbst gefälscht hast“: never believe any statistics you haven’t manipulated yourself. This German phrase tells much about people’s trust in statistics. All it needs to prove it wrong is a thorough and appropriate processing of data, avoiding crooked ways and implementing the statistical quality standards.
– manipulation: no deliberate manipulation or alteration of data, especially not in order to allegedly “find out” some “results” set in advance.
– censorship: no censoring of data, no concealment of important data or findings.
Important to keep in mind
– sample: a sample must be representative and large enough to be significant.
– exhaustiveness: all relevant data are to be processed, only the relevant data are to be processed.
– objectivity: absolute objectivity is a highly desirable ambition, but stays out of reach even for quantitative surveys.
– biases: wherever possible, identify and avoid (potential) biases. Unavoidable (structural) biases must be taken into consideration while analysing the data and be mentioned in the final report.
Processing of data sets
– control of returned questionnaires: the validity of all returned questionnaires is checked. Questionnaires with substantial apparent lack of quality as well as those filled in by individuals outside the target group are considered non-valid and taken out of the sample.
– quality control: likewise, the quality of data sets is checked. Striking errors of data entry as well as obvious, but unequivocal misunderstandings of the respondents are adjusted or corrected.
– statistical calculation: the corrected data sets are encoded and processed with a statistical software. The outcomes encompass occurrence tables and cross-tables as well as the most common key figures such as average, median or significance (chi²).
– selection: a selection has to be made out of all the (potentially) calculated statistics. The relevance and usefulness of the findings, the comparability of the variables or the correlation between them are among the most important selection criteria.
Dealing with figures
– percentages: the findings are presented in absolute numbers for surveys with small samples and in percentages for those with samples of more than 100 individuals. The calculation of percentages is mathematically correct for samples of all size, of course. In empirical social sciences however, they are considered appropriate from 100 individuals onwards, with a tolerance from 80 individuals in practice. Thus all the statistics and findings presented in percentages but calculated on a small basis (less than 80 individuals for that subset of the sample) are marked with an asterisk in the final report.
– rounding up or down: all percentages are calculated with one decimal point, which is precise enough for the range of surveys and the usual size of samples considered here. For the sake of better readability, the figures are always rounded up/down in the final report.
– presentation of findings: in the final report, all the statistics and results are either presented in form of a chart or a table, or included in the text itself. They have to be understandable, explicit and unequivocal. They should also be presented in a pleasant way so as to make the report as attractive as possible.
Keeping the data sets
Unless agreed differently with the cultural institution, all returned written questionnaires will be kept in my archive as soon as the project is completed. They will remain there for a period of 10 years, and then be disposed of.
Complete data sets of a survey, the data processing and the findings, whether digital or printed out, will likewise be kept in my archives without any limit of time. This also applies to all working documents and the final report.
Depending on the terms of agreement with the cultural institution, the written questionnaires or some parts of the data sets can be handed over (in original or copy) to the institution at the end of the project.
The rights to use the findings of an evaluation usually lie by the commissioning cultural institution. The terms of the contract often state that data sets, findings and final reports have to be handled as confidential. Thus, an explicit permission of the institution is needed before those information and findings of a survey can be shared with third persons.
However, the knowledge collected through different evaluations in the course of time is of high scientific value. It sums up to a valuable pool of benchmark data, basis or reference information, points of comparison and empirical experience which enriches the analysis and interpretation of each further evaluation. It would be absurd to ignore this treasure. Obviously, neither the cultural institution nor the project itself would be explicitly named or made recognisable – except of course if the institution gave explicit permission or if the data has already been published elsewhere.
To carry out the evaluations, I usually need the help of assistants or temporary staff. All assistants and staff joining in for a project have to some extend access to collected data, especially to the part they are assigned to process or work on. They all have to sign an agreement of confidentiality covering all the information, data and findings they might have access to during their work on the project.