Victorian Healthcare Experience Survey Methodology
The Victorian Healthcare Experience Survey (VHES) asks recent users of Victoria’s public health system to provide in-depth feedback about specific healthcare experiences.
Patient categories
Patient category refers to groups of patients, determined by their engagement with different parts of the health system. The VHES program contains surveys covering 17 patient categories and several sub-categories.
Five patient categories are surveyed continuously. Each month, invitations are sent to a sample of eligible patients discharged from care, asking them to reflect upon their experience of care in the previous month.
The remaining twelve patient categories are surveyed annually. Annual surveys follow a different data collection pattern and are tailored to the needs of the patient category being surveyed. Further details are provided below.
Continuous surveys cover the following consumer categories:
Questionnaires
Excluding the two Mental Health surveys, which utilise a national survey tool, each Questionnaire was developed locally.
The questionnaires have two sections, a Core Question Set and a Specific Patient Category- Question Set, known as Modules.
The Core Question Set aligns with the Partnering in Healthcare Framework and asks the same set of questions across all health care settings. The work was guided by an Advisory Group consisting of representatives from health services, Victorian Agency for Health Information (VAHI), Safer Care Victoria (SCV) and the Department of Health (department).
The Modules were developed in consultation with a Working Group consisting of representatives from health services, VAHI, SCV and the department.
Additional feedback on the Core Question Set and Modules was sought from health services via an online survey. Each set of questions has been cognitively tested. The Modules were tested with a minimum of 10 recent consumers from each category. The Core Questionnaire set was tested with 32 consumers who had a recent healthcare experience from a diverse range of settings.
Sampling and data collection
Multiple approaches to sampling and data collection are used, these include an in-situ and out-of-service method.
Invitations for in-situ surveys are offered to consumers during their episode of care at the health service. Invitations for out-of-service surveys are offered to patients following their episode of care.
In-situ categories are:
Out-of-service categories are:
For all out-of-service categories, consumer/carer lists for the survey are obtained directly from the campuses participating in the study. For continuous surveys this process is repeated on a monthly basis. For annual surveys it takes place over a designated time period each year. Health service representatives upload patient details using either an automated or manual process. Once all the files are uploaded, the records are passed through the process stage and a number of validation checks are undertaken. The data are cleaned prior to conducting sampling. This involves the removal of duplicate records, records sampled in the previous six months, and records with ineligible contact details.
Stratified random sampling is then conducted and involves dividing the population into sub-groups, or strata (in this case, campuses), and randomly selecting individual clients or consumers from within these strata. Dividing the sample into strata gives us greater control over the composition of the final sample and ensures the sample is proportional to the patient population.
Each selected patient is invited to the survey via an email, SMS or letter delivered to their home address, based on the contact details available. In cases where multiple contact details are available, the following hierarchy is used:
If a health service provides a patient’s preferred language and it is one of the 22 languages other than English offered by the VHES program, they will receive an invitation and survey in the preferred language. All other surveys are sent in English, however participants will have the ability to complete the survey in any one of the available languages. From October 2022, postal invitations were not sent to core survey patients, only patients with a valid email address or mobile number are considered for sampling.
Languages
All questionnaires are translated into the most commonly spoken languages among consumers of Victorian health services. These are:
Reporting and Analysis
Weighting of results
To ensure that data accurately represents the population of interest (the true population of patients at each campus), normalising factors are applied to the data. Normalising factors are calculated based on the difference between the proportion of a certain type of respondent in the sample and the proportion of that type of respondent attending the health service. A proportional weighting scheme is applied to the sample at each campus to bring it into line with each campus's true population by age and gender. This scheme is based on the patient data uploaded by each campus and applies to all adult inpatient, emergency, ambulance and specialist clinics results. This means, for example, that older people aren't over-represented in the survey results by virtue of being more likely to participate in the survey than younger people. No age or gender weighting is applied to paediatric, maternity, palliative care or community health results.
When looking at results that reflect a group of campuses (for example, at health service, state or peer group level) each campus is normalised according to what percentage of the group's true population it represents. This means, for example, that one hospital isn't underrepresented because its patients were less likely to complete the survey than those at another hospital. This level of normalisation occurs for all patient categories.
Positive response coding
Each questionnaire contains questions that, in analysis, are re-coded to represent a ‘Positive’ score. This ‘Positive’ score is used for the time-series reporting) and for comparison.
Experience questions are typically administered on a three or four-point scale including a ‘Yes, always’ (or similar) response, a ‘Yes, sometimes’ (or similar) response and a ‘No’ response. At times this scale includes not applicable options, such as a ‘I’m not sure’, ‘I did not want or need this’ (or similar) or ‘Don’t know’ response.
In most cases, the ‘Positive’ score is the single most desirable response (e.g. ‘Yes, definitely’). The exception to this rule is the overall experience question, in this instance the ‘Very Good’ and ‘Good’ are considered a positive response. Neutral answers are excluded from the denominator when calculating the positive response score, and are noted for each question on the detailed results page.
Significance testing
Significance testing has been conducted throughout the results portal. Significance testing provides a simple and meaningful way of identifying whether differences observed in sample data are likely to reflect true differences in the population. However, we always emphasise that a ‘difference’ does not necessarily mean ‘importance’.
Throughout all VHES reporting, a z test is used to calculate significance. A z-test is a statistical test used to determine whether two population means are different when the variances are known and the sample size is large.
Statistically relevant significant findings have been reported at the 95% confidence interval and are represented against the comparator (state average, peer group average, health service average or previous result) with a green dot where the subject (campus, health region or state) has performed significantly higher than the comparator or with an red dot where it has performed significantly lower. The same test is applied when comparing results within a subject over time.
Confidence Intervals
A confidence level is a description of how similar the sample is to the entire population, when the sample is selected at random. It is expressed as a percentage. We use a 95% confidence level, which means that we expect the characteristics of the sample to be the same as the whole population 95 times out of 100.
Confidence intervals are reported in the VHES results portal as bands on all time-series charts. Users can also view the upper and lower confidence interval by hovering over a data point on the time series.
Minimum sample size requirement
For all categories, results for each campus or health service are published on the portal only if the campus achieves the minimum number of survey returns requirement to ensure the data is statistically robust. However, these campuses will still contribute to the region, peer group, health service and state average. Where a campus does not achieve this minimum survey returns, the portal will state that there is insufficient data for this period.
Campuses are required to achieve 10 completed surveys to have their results published on the portal. This requirement applies to all patient categories.
However, campuses who do not receive the minimum will receive a rolling sample as the survey progresses. This means that results for multiple quarters will be merged to provide a sufficient sample size over a longer period. This is called ‘accumulative reporting’ and is only implemented at campus level.
Provided a level has met the minimum overall sample size of n=10, all questions will be reported so long as one response has been achieved. However, an alert symbol will be shown where the sample size for a particular question is less than 30. This means that the margin of error for this question is quite large (more than +/- 18%), so results should be interpreted with caution. When disaggregating results at ward-level: if a ward does not achieve 10 survey responses, the portal states that there is insufficient data for this period. Responses from wards that do not achieve 10 survey returns will to still contribute to campus results.
The Victorian Healthcare Experience Survey (VHES) asks recent users of Victoria’s public health system to provide in-depth feedback about specific healthcare experiences.
Patient categories
Patient category refers to groups of patients, determined by their engagement with different parts of the health system. The VHES program contains surveys covering 17 patient categories and several sub-categories.
Five patient categories are surveyed continuously. Each month, invitations are sent to a sample of eligible patients discharged from care, asking them to reflect upon their experience of care in the previous month.
The remaining twelve patient categories are surveyed annually. Annual surveys follow a different data collection pattern and are tailored to the needs of the patient category being surveyed. Further details are provided below.
Continuous surveys cover the following consumer categories:
- Adult Inpatients aged 16 or over who were admitted to a public hospital
- Adult Emergency patients aged 16 or over who attended a public hospital emergency department but were not admitted.
- Maternity consumers aged 16 or over who received services from a public hospital in relation to antenatal, labour, birth and/or postnatal care.
- Paediatric Inpatients aged 15 or under who were admitted to a public hospital and/or their parents and carers.
- Paediatric Emergency patients aged 15 or under who attended a public hospital emergency department but were not admitted and/or their parents and carers.
- Community Health: Adult and paediatric clients who used public community health services
- Palliative Care Bereaved Carer: Bereaved carers of palliative care clients who used public hospital (admitted) or in-home (community) services
- Palliative Care Client: Palliative care clients who used public hospital (admitted) or in-home (community) services
- Palliative Carer Current Carer: Carers of clients in palliative care who used public hospital (admitted) or in-home (community) services
- Planned Ambulance: Users of planned ambulance service
- Emergency Ambulance: Users of emergency ambulance services
- Your Experience of Service (Mental Health): Users of public mental health services.
- Mental Health Carer Experience Survey: Carers of clients who have used public mental health services.
- Adult Specialist Clinics: Adult specialist clinics patients who had an outpatient appointment at public hospitals offering these services.
- Paediatric Specialist Clinics: Paediatric specialist clinics patients who had an outpatient appointment at public hospitals offering these services.
- Cancer Survey: Patients treated for cancer at a public hospital
- Home and Community Care Program for Your People: Adult and child clients who accessed services provided by participating services.
Questionnaires
Excluding the two Mental Health surveys, which utilise a national survey tool, each Questionnaire was developed locally.
The questionnaires have two sections, a Core Question Set and a Specific Patient Category- Question Set, known as Modules.
The Core Question Set aligns with the Partnering in Healthcare Framework and asks the same set of questions across all health care settings. The work was guided by an Advisory Group consisting of representatives from health services, Victorian Agency for Health Information (VAHI), Safer Care Victoria (SCV) and the Department of Health (department).
The Modules were developed in consultation with a Working Group consisting of representatives from health services, VAHI, SCV and the department.
Additional feedback on the Core Question Set and Modules was sought from health services via an online survey. Each set of questions has been cognitively tested. The Modules were tested with a minimum of 10 recent consumers from each category. The Core Questionnaire set was tested with 32 consumers who had a recent healthcare experience from a diverse range of settings.
Sampling and data collection
Multiple approaches to sampling and data collection are used, these include an in-situ and out-of-service method.
Invitations for in-situ surveys are offered to consumers during their episode of care at the health service. Invitations for out-of-service surveys are offered to patients following their episode of care.
In-situ categories are:
- Your Experience of Service (Mental Health)
- Palliative Care Hospital Client and Carer
- Community Health (in 2021)
Out-of-service categories are:
- Adult Inpatient
- Adult Emergency
- Maternity
- Paediatric Inpatient
- Paediatric Emergency Patient
- Palliative Care Bereaved Carer
- Palliative care in-home (community) Client and Carer
- Planned Ambulance
- Emergency Ambulance
- Mental Health Carer Experience Survey
- Adult Specialist Clinics
- Paediatric Specialist Clinics
- Community Health (2023)
- HACC PYP
- Cancer
For all out-of-service categories, consumer/carer lists for the survey are obtained directly from the campuses participating in the study. For continuous surveys this process is repeated on a monthly basis. For annual surveys it takes place over a designated time period each year. Health service representatives upload patient details using either an automated or manual process. Once all the files are uploaded, the records are passed through the process stage and a number of validation checks are undertaken. The data are cleaned prior to conducting sampling. This involves the removal of duplicate records, records sampled in the previous six months, and records with ineligible contact details.
Stratified random sampling is then conducted and involves dividing the population into sub-groups, or strata (in this case, campuses), and randomly selecting individual clients or consumers from within these strata. Dividing the sample into strata gives us greater control over the composition of the final sample and ensures the sample is proportional to the patient population.
Each selected patient is invited to the survey via an email, SMS or letter delivered to their home address, based on the contact details available. In cases where multiple contact details are available, the following hierarchy is used:
- SMS
- Posted letter
If a health service provides a patient’s preferred language and it is one of the 22 languages other than English offered by the VHES program, they will receive an invitation and survey in the preferred language. All other surveys are sent in English, however participants will have the ability to complete the survey in any one of the available languages. From October 2022, postal invitations were not sent to core survey patients, only patients with a valid email address or mobile number are considered for sampling.
Languages
All questionnaires are translated into the most commonly spoken languages among consumers of Victorian health services. These are:
- Amharic
- Arabic
- Burmese
- Chinese Simplified
- Chinese Traditional
- Croatian
- Dari
- English
- Greek
- Hindi
- Italian
- Khmer
- Macedonian
- Maltese
- Oromo
- Persian
- Polish
- Russian
- Serbian
- Spanish
- Tibetan
- Turkish
- Vietnamese
Reporting and Analysis
Weighting of results
To ensure that data accurately represents the population of interest (the true population of patients at each campus), normalising factors are applied to the data. Normalising factors are calculated based on the difference between the proportion of a certain type of respondent in the sample and the proportion of that type of respondent attending the health service. A proportional weighting scheme is applied to the sample at each campus to bring it into line with each campus's true population by age and gender. This scheme is based on the patient data uploaded by each campus and applies to all adult inpatient, emergency, ambulance and specialist clinics results. This means, for example, that older people aren't over-represented in the survey results by virtue of being more likely to participate in the survey than younger people. No age or gender weighting is applied to paediatric, maternity, palliative care or community health results.
When looking at results that reflect a group of campuses (for example, at health service, state or peer group level) each campus is normalised according to what percentage of the group's true population it represents. This means, for example, that one hospital isn't underrepresented because its patients were less likely to complete the survey than those at another hospital. This level of normalisation occurs for all patient categories.
Positive response coding
Each questionnaire contains questions that, in analysis, are re-coded to represent a ‘Positive’ score. This ‘Positive’ score is used for the time-series reporting) and for comparison.
Experience questions are typically administered on a three or four-point scale including a ‘Yes, always’ (or similar) response, a ‘Yes, sometimes’ (or similar) response and a ‘No’ response. At times this scale includes not applicable options, such as a ‘I’m not sure’, ‘I did not want or need this’ (or similar) or ‘Don’t know’ response.
In most cases, the ‘Positive’ score is the single most desirable response (e.g. ‘Yes, definitely’). The exception to this rule is the overall experience question, in this instance the ‘Very Good’ and ‘Good’ are considered a positive response. Neutral answers are excluded from the denominator when calculating the positive response score, and are noted for each question on the detailed results page.
Significance testing
Significance testing has been conducted throughout the results portal. Significance testing provides a simple and meaningful way of identifying whether differences observed in sample data are likely to reflect true differences in the population. However, we always emphasise that a ‘difference’ does not necessarily mean ‘importance’.
Throughout all VHES reporting, a z test is used to calculate significance. A z-test is a statistical test used to determine whether two population means are different when the variances are known and the sample size is large.
Statistically relevant significant findings have been reported at the 95% confidence interval and are represented against the comparator (state average, peer group average, health service average or previous result) with a green dot where the subject (campus, health region or state) has performed significantly higher than the comparator or with an red dot where it has performed significantly lower. The same test is applied when comparing results within a subject over time.
Confidence Intervals
A confidence level is a description of how similar the sample is to the entire population, when the sample is selected at random. It is expressed as a percentage. We use a 95% confidence level, which means that we expect the characteristics of the sample to be the same as the whole population 95 times out of 100.
Confidence intervals are reported in the VHES results portal as bands on all time-series charts. Users can also view the upper and lower confidence interval by hovering over a data point on the time series.
Minimum sample size requirement
For all categories, results for each campus or health service are published on the portal only if the campus achieves the minimum number of survey returns requirement to ensure the data is statistically robust. However, these campuses will still contribute to the region, peer group, health service and state average. Where a campus does not achieve this minimum survey returns, the portal will state that there is insufficient data for this period.
Campuses are required to achieve 10 completed surveys to have their results published on the portal. This requirement applies to all patient categories.
However, campuses who do not receive the minimum will receive a rolling sample as the survey progresses. This means that results for multiple quarters will be merged to provide a sufficient sample size over a longer period. This is called ‘accumulative reporting’ and is only implemented at campus level.
Provided a level has met the minimum overall sample size of n=10, all questions will be reported so long as one response has been achieved. However, an alert symbol will be shown where the sample size for a particular question is less than 30. This means that the margin of error for this question is quite large (more than +/- 18%), so results should be interpreted with caution. When disaggregating results at ward-level: if a ward does not achieve 10 survey responses, the portal states that there is insufficient data for this period. Responses from wards that do not achieve 10 survey returns will to still contribute to campus results.
