Nordic Digital Health Evaluation Criteria

Created by
Last updated 17 April 2023, 15:36

Watch for changes

✓ Subscribed

 

Introduction

The Nordic Digital Health & Medication Platform project aims to establish a system for healthcare providers to evaluate and identify trusted digital health technologies within healthcare and preventive care. These digital health technologies will be evaluated by the Nordic Digital Health Evaluation Criteria (NordDEC). Below you will find the scoring and questions that make up the criteria.

Value and Risk Points

  • The scoring is made up of Value earning points and Risk earning points.

  • Each scoring question has either a Risk implication or a Value implication

  • The quantum of the Risk or Value implication is decided by the relevant tariff which range from small, medium, high or exceptionally high in the Risk area and small, medium or high in the Value area.

  • The following table sets out the actual numeric value of each Tariff:

Tariff

Risk

Value

Small

10

5

Medium

20

10

High

40

20

Exceptionally High

80

-

  • In addition to the base tariff, some risk and value related questions attract a multiplier that will increase the relevant tariff based on certain related app characteristics.

  • Maximum risk can be applied based on responses for certain questions. Maximum risk is applied to a whole section (i.e Data), rather than an individual question. It is the sum of all the risk points that could be applied if were not for the questions being disabled by earlier responses.

SCENE SETTERS

The NordDEC begins with a series of questions designed to assess core purpose and functionality of digital health products in the form of native apps or web apps (“apps”). For the purpose of the NordDEC a digital health app is defined as “a digitally delivered product that is aimed at supporting in some way general health or specific conditions”. The questions look to capture the target audience, the type of data the app collects and the apps primary functions and features. None of the scene setter questions are intended to have any scoring or risk implications and are purely to decide on the line of enquiry further in the evaluation.

Every question within scene setters does not have a scoring value.

App Characteristics

Question

Question Reference Source

Is the App health focused?

ORC_SS01

 Further Information

Guidance/Context

The purpose of this question is to identify apps which are within the NordDEC scope of assessment. This includes any apps which have a clear health or medical purpose, are condition specific, or have a valid place in a clinical setting. This also includes wellness apps if they have a clear focus on a particular need or condition, eg. yoga apps for pregnancy. Apps which have no clear or specific health focus are excluded, eg. generic meditation apps.

Response

Yes / No

Answer Criteria

Yes: If the app has a specific health, fitness, lifestyle purpose or claim. If the app is condition specific. If the app has a clear place in a clinical setting.

No: If app has no obvious health purpose (e.g. voice recorder, screen recorder, keyboard, a timer app, recipe books)If the app has no health purpose and does not relate to any kind of health condition (e.g. general meditation apps). Fitness app where exercises are not designed to prevent a specific condition e.g. a circuit training app with no health claims.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Data - Data Types, Data Collection and Data Sharing

Question

Question Reference Source

Does the App collect data? 

ORC_D01

 Further Information

Guidance/Context

The purpose of this question is to identify if the app collects data so the relevant data questions are disabled appropriately.

Response

Yes/No

Answer Criteria

Yes: If any data is collected by or through the app, in any way. Including data such as usage data, cookies etc.

No: If no data is collected from the user or the app.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

What type of data is collected by the App?

ORC_DT10

 Further Information

Guidance/Context

This question aims to identify what type of data the app collects. This is answered based on what information can be submitted into the app and also what is visible in the privacy policy.
To select data items from the privacy policy it must be clear that the data items are are collected via the app and not via an associated website. For instance if the privacy policy states “when you use our services…”, it is essential that the term ‘services’ is checked to see if it is referring to the app or not. If ‘services’ does refer to the app then the data items mentioned following that statement should be selected but if ‘services’ simply refers to an associated website then the data items should not be selected.
If cookies are mentioned in the privacy policy, they are only included if they relate to the app. Biometric data is only selected if such data is directly processed by the app and/or the device’s inbuilt software isn’t used.

Response

Multiple Choice

Answer Criteria

Name (Full Name, Nickname or First Name Only), Username, Mobile Number/Device Number/Home Phone Number, Email, Full Address/Postcode, Age/DOB, Location Data, IP Address, Device IMEI Number, Other Unique Device Identifiers, General Wellness Data, General Identifier e.g. NHS No, Physical Description, Martial Status/Family/Lifestyle/Social Circumstance, Income/Financial/Tax Situation, Education/Qualifications/Professional Training/Awards, Employment/Career History, Cookies/Web Beacons etc. (used for tracking an individual’s online browsing behaviours/movements), Usage Data, Card/Payment/Financial Information, Race/Ethnic Origin, Political/Religious/Philosophical Beliefs or Opinions, Trade Union Membership, Genetic or Biometric Data (where this is used for identification purposes e.g. fingerprint or facial recognition), Physical and/or Mental Health Data, Sexual Orientation or Sex Life, Gender (self-declared or observed), Offences Committed/Alleged to have Committed/Criminal Proceedings/Outcomes/Sentence, Other Online Identifiers, Other (please specify).

Logic

DISABLEMENT LOGIC - Disabled if D01 is no.

Scoring Impact

There is no scoring impact associated with this question.

What Permissions does the app request?

ORC_ERC_OTS_P01

 Further Information

Guidance/Context

This question is only relevant if the platform is a mobile app. You can find the answer to this question through the device’s settings and through the google play store and app store. This question helps inform the Technical Security questions which are asked later in the assessment.

Response

Free text

Answer criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Are users required/able to sign up/register to use the service?

ORC_DT14

 Further Information

Guidance/Context

If a user is required or able to sign up to use the app, it indicates that personal information is undoubtedly collected and processed as part of the service. The assessor may then have to determine whether or not the collection and processing of personal information is strictly necessary for the provision of services.

Response

Yes / No

Answer Criteria

Yes: If any part of the service provided requires a user to set up an account.

Yes: If account creation is not mandatory, but is optional for the purpose of backing up information.

No: If account creation is not possible in any circumstance.

Logic

DISABLEMENT LOGIC - Disabled if D01 is no.

Scoring Impact

There is no scoring impact associated with this question.

Is data collected through cookies?

ORC_DT11

 Further Information

Guidance/Context

This question is pre-filled if 'cookies/web beacons etc.' has been selected in DT10. This will require validation from the assessor. To answer yes, any cookies mentioned in the privacy policy/cookie policy must be in reference to the relevant app and not the associated website. This question determines whether the following cookie questions will be asked throughout the assessment.

Response

Yes / No

Answer Criteria

Yes: If the privacy policy/cookie policy states the application uses cookies.

Yes: If the application stops functioning when cookies are disabled through the device settings.

No: If there are no mention of cookies in the app or on the privacy policy.

No: If cookies are only mentioned in relation to the associated website.

Logic

DISABLEMENT LOGIC - Disabled if D01 is no OR if DT10 does not contain cookies/web beacons etc.

Scoring Impact

There is no scoring impact associated with this question.

What type of cookies are used?

ORC_DT12

 Further Information

Guidance/Context

The type of cookies hold different level of importance on what rights must be upheld for the user and can also act as indicators as to the nature of the data collected through cookies and whether “profiling” might be occurring, making a user identifiable.
Third-party (aka tracking) cookies are used to collect data based on online behaviour. This data is passed on/sold to third-party advertisers so that the information can be used for targeted adverts. User’s must be given the option to block these.
Session cookies track movement around a website and can be ‘strictly necessary’. These expire as soon as you leave/close the website. An example of why these might be strictly necessary would be for online shopping, if session cookies aren’t used, then when adding to a cart and ultimately going to check-out, your cart would appear empty.
Persistent/permanent cookies are used for remembering and implementing user preferences for when they return to a website. These cookies are stored on your hard disk for extended periods of time and will have varying expiration dates. Once deleted, everything customised for preferences will be forgotten. Persistent cookies are often used for computers to remember and store your login information, language selections, menu preferences, etc.
If it is unclear, even through a cookie policy, as to the types of cookie being used, this might be used to apply immediate negative scoring on apps being asked questions regarding cookies in the data section.

Response

Multiple Choice

Answer Criteria

Third party: If the privacy policy or cookie policy states that third party cookies are used.

Session: If the privacy policy or cookie policy states that third party cookies are used.

Persistent: If the privacy policy or cookie policy states that third party cookies are used.

Unclear: If the privacy policy or cookie policy is unclear to the user what types of cookies are being used.

Answer Criteria

Logic

DISABLEMENT LOGIC - Disabled if D01 is no OR if DT10 does not contain cookies/web beacons etc OR DT11 equals no.

Scoring Impact

There is no scoring impact associated with this question.

Is the data (cookie and/or none cookie) collected:

ORC_DT13

 Further Information

Guidance/Context

This question aims to determine what level the data is classified as i.e. sensitive, personal or nonpersonal. The level of data impacts what level of rights should be upheld for the user. The assessor will select the appropriate level of data by referring to the data they selected in DT10. ‘Personal’ data relates to data which can be used to identify someone whereas ‘Personal (combined)’ data refers to a number of pieces of data which when combined can be used to identify someone.

Response

Multiple Choice

Answer Criteria

Sensitive: Physical / Mental Health or Condition (past, current or future status), Sexual Life / Orientation, Political, Religious or other beliefs or opinions, Offences Committed / Alleged to have Committed / Criminal Proceedings / Outcomes / Sentence,  Financial data (that might be used for payment fraud), Trade Union membership, Racial / Ethnic Origin, Genetic or Biometric Data (e.g. fingerprints / facial Recognition) for the purpose of uniquely identifying a person

Personal (combined): Cookies, web beacons, flash cookies, server logs etc which track individual’s browsing behaviour, Other Unique Device Identifiers eg. Device MAC Address, Name, Age/DOB, Gender (self declared or observed), Marital Status |Family / Lifestyle / Social Circumstance, Education / Qualifications / Professional Training / Awards, Other online identifiers / Event Logs, Location Data (Travel / GPS / GSM Data / radio frequency identification tags (RFID))

Personal: Address/Postcode (full), Email Address, Mobile Phone Number / Device Number / Home Phone Number, Physical Description, Username, IP Address, General Identifier e.g. Social Security Number, Income / Financial / Tax Situation, Employment / Career History, Device IMEI No

Non-personal: General Wellness data

Logic

DISABLEMENT LOGIC - Disabled if D01 is no.

Scoring Impact

There is no scoring impact associated with this question.

How is non-cookie data collected?

ORC_DC01

 Further Information

Guidance/Context

This question aims to uncover how data is collected from an individual, this information is key to other parts of the assessment. It is particularly important to ensure that organisations make it clear to their users when there is any “blind” processing occurring.

Response

Multiple Choice

Answer Criteria

Device measurement capability: Auto GPS, motion, microphone, camera

Other apps: Google fit, Apple health, Facebook

Devices: Wearables, Medical devices

Third party sources: Google analytics, card payment processors (stripe, PayPal)

Automatically generated by the app: Usage data

From Device storage: Photos saved on device

From Device Information: IP address

Other (please specify: Assessor to specify what other is in the comment box

Logic

DISABLEMENT LOGIC - Disabled if D01 is no.

Scoring Impact

There is no scoring impact associated with this question.

What other apps is the App connected to?

ORC_DC02

 Further Information

Guidance/Context

Integration of information between different apps/platforms can be of value to certain individuals. That said, it can also present additional security risks. The organisation should consider and mitigate these security risks when they have enabled personal information to be shared to and from their product.

Response

Free Text

Answer Criteria

N/A

Logic

DISABLEMENT LOGIC - Disabled if D01 is no OR if DC01 does not contain ‘other apps’.

Scoring Impact

There is no scoring impact associated with this question.

What device(s) does the App connect to? 

ORC_DC03

 Further Information

Guidance/Context

Integration of information from devices can be of value to certain individuals. That said, it can also present additional security risks. The organisation should consider and mitigate these security risks when they have enabled personal information to be shared to and from their product.

Response

Free Text

Answer Criteria

N/A

Logic

DISABLEMENT LOGIC - Disabled if D01 is no OR if DC01 does not contain ‘other devices’.

Scoring Impact

There is no scoring impact associated with this question.

Can the user prevent cookie data being collected and still use the App?

ORC_DS01

 Further Information

Guidance/Context

Under the Privacy directive any organisation must minimise access restrictions, even if a user refuses to accept certain cookies.

Response

Yes / No

Answer Criteria

Yes: If the assessor has been able to leave the app > go to device browser > identify that cookies are in use > turn off all cooking relating to the service > launch and access the app again > check back on browser to ensure the previously removed/blocked cookies have not become active again.

Yes: If the user is informed that they can prevent cookies and that this will only possibly prevent some functionality/access to features.

No: If cookies have become active again on the device’s browser after following the steps to turn them off.

No: If strictly necessary cookies are in use.

No: If users are not given the option or informed how to control/prevent/turn off cookies.

Logic

DISABLEMENT LOGIC - Disabled if D01 is no OR if DC01 does not contain 'cookies/web beacons etc’.

Scoring Impact

There is no scoring impact associated with this question.

Does the disabling of cookies impact the use of the App in any way?

ORC_DS02

 Further Information

Guidance/Context

This helps to identify whether cookies are necessary for the app to function.

Response

Yes / No

Answer Criteria

Yes: If the assessor has been able to leave the app > go to device browser > identify that cookies are in use > turn off all cooking relating to the service > launch and access the app again > has been unable to access certain features.

Yes: If the user is informed that they can prevent cookies and that this will only possibly prevent some functionality/access to features.

No: If disabling the cookies through the browser has no impact on functionality/access to features.

No: If strictly necessary cookies are in use.

Logic

DISABLEMENT LOGIC - Disabled if D01 is no OR if DC01 does not contain 'cookies/web beacons etc’.

Scoring Impact

There is no scoring impact associated with this question.

Can/is data shared? (excluding cookies)

ORC_DS03

 Further Information

Guidance/Context

This question determines whether the following data sharing questions are asked.

Response

Yes / No

Answer Criteria

Yes: If any data type that has been identified as collected is shared/exported from the App, on the device, in any way. This includes data being transferred and stored by the developer on external servers and includes the ability for the user to manually move data out of the app.

Yes: If you have to create an account to access the app.

No: If there is no data transferred from the app, to another location, on or off the device, either automatically or by manual export by the user.

Logic

DISABLEMENT LOGIC - Disabled if D01 is no.

Scoring Impact

There is no scoring impact associated with this question.

Can data be shared through a direct, manual action by the user? {e.g. by sending data via email or manually choosing to post/share something within the app etc}

ORC_DS04

 Further Information

Guidance/Context

This question helps inform the user if they are able to share their own health data via the app.

Response

Yes / No

Answer Criteria

Yes: If any data only leaves the app or the device when the user carries out a direct action for this to occur. This action needs to be carried out every time the user wishes to share this data. (sharing data via email or sending reports manually within the app).

Yes: Manually choosing to post/share something within the app.

No: If data is shared without a direct action from the user.

No:  If data is automatically transferred following a single action of turning on a permission in the app.

Logic

DISABLEMENT LOGIC - Disabled if D01 is no OR if DS03 is no.

Scoring Impact

There is no scoring impact associated with this question.

How is the user able to manually share their data?

ORC_DS05

 Further Information

Guidance/Context

This question helps inform the user how exact they are able to share their health data.

Response

Multiple Choice

Answer Criteria

Exporting a report to the device: If the user can store data, outside of the app, on the device itself.

Exporting data to a preset email: If the user has the option to email reports/data and is taken to the device email app to send information.

Exporting data to messaging services: If the user has the option to share information/data and the share options include any messaging services/apps on the device. Emails do not count as messaging services.

Transfer through Bluetooth: If the user can manually share information using the devices Bluetooth. This does not include automatic transfer of data to a device that automatically connects. Nor does it include apps that continuously run Bluetooth in the background to communicate and recognise other devices (such as track and trace).

Transfer through NFC (near-field communication): If the user can transfer data to another device using NFC capabilities.

Manually choose to post/share something in the app: If there is any form of in app communication between two or more users, where the information/content posted is done so through the user opting to do so each time.

Other (please specify): If the app is transferring through manual user intervention but the option has not been listed.

Logic

DISABLEMENT LOGIC - Disabled if D01 is no OR if DS04 is no.

Scoring Impact

There is no scoring impact associated with this question.

Is data ONLY shareable through a direct, manual action by the user? (excluding cookies)

ORC_DS06

 Further Information

Guidance/Context

If data on the app is only manually shared, the app with external third parties of the user’s choice and is often something that is not managed or decided by the developer. Therefore if this is the only form of data sharing, and the developer cannot access or process the data away from the device that the app is on, they will be less likely to be subject to GDPR principals.

Response

Yes / No

Answer Criteria

Yes: If the only data transfers that occur are through the direct user interactions identified.

No: If there is any data transferred by a means that has not been done through a direct user intervention. For example if you have selected “usage data” and this is collected automatically, then you would answer No.

Logic

DISABLEMENT LOGIC - Disabled if D01 is no OR if DS04 is no.

Scoring Impact

There is no scoring impact associated with this question.

Can the user control any automatic data sharing, through setting individual sharing preferences in the app? (excluding cookies)

ORC_DS07

 Further Information

Guidance/Context

This question aims to find out whether users can set up automatic data sharing. Signing up through Facebook does not count, the user has to have control for example, toggle something in the app to choose ‘yes you can collect usage data’.

Response

Yes / No

Answer Criteria

Yes: If the user has control over when data is automatically shared, for example, through having individual options that can be toggled on and off, in the app.

Yes: If the user can create social circles, or choose who can view/access their profile/data. Eg. Making an account private or public, or specifically selecting which members of your clinical support network can view which data.

No: If the user has no choice in whether or not to sign up to the app in order to use it, i.e. if the user MUST sign up to use, or CANNOT sign up at all.

Logic

DISABLEMENT LOGIC - Disabled if D01 is no OR if DS03 is no OR if DS06 is yes.

Scoring Impact

There is no scoring impact associated with this question.

Where/With who can the user share data automatically by manually setting sharing preferences in the app?

ORC_DS08

 Further Information

Guidance/Context

If the user can control automatic data sharing via a toggle, this question identifies with whom the user can share the data.

Response

Multiple Choice

Answer Criteria

Developer

Clinician/HCP

Other users 

Third parties: Google Fit, Apple Health, Facebook, google analytics, etc.

Other devices: Wearables, scales, medical devices.

Logic

DISABLEMENT LOGIC - Disabled if D01 is no OR if DS07 is no.

Scoring Impact

There is no scoring impact associated with this question.

Is any data (excluding cookie data) shared automatically as soon as the App is accessed – based only on agreement to relevant Terms of Use or Privacy Policy?

ORC_DS09

 Further Information

Guidance/Context

This question aims to identify if any automatic sharing of data occurs without any input from the user beyond agreeing to the Privacy Policy and/or T&C’s.

Response

Yes / No

Answer Criteria

Yes: The privacy policy states that data is automatically shared.

No: The privacy policy / app clearly states that no data is shared without the users input.

Logic

DISABLEMENT LOGIC - Disabled if D01 is no OR if DS03 is no or IF DS06 is yes.

Scoring Impact

There is no scoring impact associated with this question.

Where/With who is data automatically shared - based only on user agreement to the developer’s Privacy Policy and/or Terms of Use?

ORC_DS10

 Further Information

Guidance/Context

This question highlights with who the user’s data is automatically shared with.

Response

Multiple Choice

Answer Criteria

Developer,

Clinician/HCP

Other users

Third parties: Google Fit, Apple Health, Facebook, google analytics, etc.

Other devices: Wearables, scales, medical devices.

Logic

DISABLEMENT LOGIC - Disabled if D01 is no OR if DS09 is no.

Scoring Impact

There is no scoring impact associated with this question.

What data is automatically shared with the developer?

ORC_DS12

 Further Information

Guidance/Context

If the policy has clearly stated that data is automatically shared with the developer but is not clear exactly what data is shared then assessors can assume that all collected data is automatically shared. Do not select Biometric data if it is using the phone's native scanner/face ID rather than an in-app one scanner.

Response

Multiple Choice

Answer Criteria

Name (Full Name, Nickname or First Name Only), Username, Mobile Number/Device Number/Home Phone Number, Email, Full Address/Postcode, Age/DOB, Location Data, IP Address, Device IMEI Number, Other Unique Device Identifiers, General Wellness Data, General Identifier e.g. NHS No, Physical Description, Martial Status/Family/Lifestyle/Social Circumstance, Income/Financial/Tax Situation, Education/Qualifications/Professional Training/Awards, Employment/Career History, Cookies/Web Beacons etc. (used for tracking an individual’s online browsing behaviours/movements), Usage Data, Card/Payment/Financial Information, Race/Ethnic Origin, Political/Religious/Philosophical Beliefs or Opinions, Trade Union Membership, Genetic or Biometric Data (where this is used for identification purposes e.g. fingerprint or facial recognition), Physical and/or Mental Health Data, Sexual Orientation or Sex Life, Gender (self-declared or observed), Offences Committed/Alleged to have Committed/Criminal Proceedings/Outcomes/Sentence, Other Online Identifiers, Other (please specify).

Logic

DISABLEMENT LOGIC - Disabled if D01 is no OR if DS06 is yes. OR if DS08 does not contain developer AND DS10 does not contain developer.

Scoring Impact

There is no scoring impact associated with this question.

What data is automatically shared with physicians / healthcare professionals?

ORC_DS13

 Further Information

Guidance/Context

If the policy has clearly stated that data is automatically shared with the physicians / healthcare professionals but is not clear exactly what data is shared then assessors can assume that all collected data is automatically shared. Do not select Biometric data if it is using the phone's native scanner/face ID rather than an in-app one scanner.

Response

Multiple Choice

Answer Criteria

Name (Full Name, Nickname or First Name Only), Username, Mobile Number/Device Number/Home Phone Number, Email, Full Address/Postcode, Age/DOB, Location Data, IP Address, Device IMEI Number, Other Unique Device Identifiers, General Wellness Data, General Identifier e.g. NHS No, Physical Description, Martial Status/Family/Lifestyle/Social Circumstance, Income/Financial/Tax Situation, Education/Qualifications/Professional Training/Awards, Employment/Career History, Cookies/Web Beacons etc. (used for tracking an individual’s online browsing behaviours/movements), Usage Data, Card/Payment/Financial Information, Race/Ethnic Origin, Political/Religious/Philosophical Beliefs or Opinions, Trade Union Membership, Genetic or Biometric Data (where this is used for identification purposes e.g. fingerprint or facial recognition), Physical and/or Mental Health Data, Sexual Orientation or Sex Life, Gender (self-declared or observed), Offences Committed/Alleged to have Committed/Criminal Proceedings/Outcomes/Sentence, Other Online Identifiers, Other (please specify).

Logic

DISABLEMENT LOGIC - Disabled if D01 is no OR if DS06 is yes. OR if DS08 does not contain physician/healthcare professional AND DS10 does not contain physician/healthcare professional.

Scoring Impact

There is no scoring impact associated with this question.

What data is automatically shared with other users?

ORC_DS14

 Further Information

Guidance/Context

If the policy has clearly stated that data is automatically shared with other users but is not clear exactly what data is shared and assessors are unable tp infer this from using the app, it should be assumed that all collected data is automatically shared. Do not select Biometric data if it is using the phone's native scanner/face ID rather than an in-app one scanner.

Response

Multiple Choice

Answer Criteria

Name (Full Name, Nickname or First Name Only), Username, Mobile Number/Device Number/Home Phone Number, Email, Full Address/Postcode, Age/DOB, Location Data, IP Address, Device IMEI Number, Other Unique Device Identifiers, General Wellness Data, General Identifier e.g. NHS No, Physical Description, Martial Status/Family/Lifestyle/Social Circumstance, Income/Financial/Tax Situation, Education/Qualifications/Professional Training/Awards, Employment/Career History, Cookies/Web Beacons etc. (used for tracking an individual’s online browsing behaviours/movements), Usage Data, Card/Payment/Financial Information, Race/Ethnic Origin, Political/Religious/Philosophical Beliefs or Opinions, Trade Union Membership, Genetic or Biometric Data (where this is used for identification purposes e.g. fingerprint or facial recognition), Physical and/or Mental Health Data, Sexual Orientation or Sex Life, Gender (self-declared or observed), Offences Committed/Alleged to have Committed/Criminal Proceedings/Outcomes/Sentence, Other Online Identifiers, Other (please specify).

Logic

DISABLEMENT LOGIC - Disabled if D01 is no OR if DS06 is yes. OR if DS08 does not contain other users AND DS10 does not contain other users.

Scoring Impact

There is no scoring impact associated with this question.

What data is automatically shared with third parties?

ORC_DS15

 Further Information

Guidance/Context

If the policy has clearly stated that data is automatically shared with third parties but is not clear exactly what data is shared then assessors can assume that all collected data is automatically shared. Do not select Biometric data if it is using the phone's native scanner/face ID rather than an in-app one scanner.

Response

Multiple Choice

Answer Criteria

Name (Full Name, Nickname or First Name Only), Username, Mobile Number/Device Number/Home Phone Number, Email, Full Address/Postcode, Age/DOB, Location Data, IP Address, Device IMEI Number, Other Unique Device Identifiers, General Wellness Data, General Identifier e.g. NHS No, Physical Description, Martial Status/Family/Lifestyle/Social Circumstance, Income/Financial/Tax Situation, Education/Qualifications/Professional Training/Awards, Employment/Career History, Cookies/Web Beacons etc. (used for tracking an individual’s online browsing behaviours/movements), Usage Data, Card/Payment/Financial Information, Race/Ethnic Origin, Political/Religious/Philosophical Beliefs or Opinions, Trade Union Membership, Genetic or Biometric Data (where this is used for identification purposes e.g. fingerprint or facial recognition), Physical and/or Mental Health Data, Sexual Orientation or Sex Life, Gender (self-declared or observed), Offences Committed/Alleged to have Committed/Criminal Proceedings/Outcomes/Sentence, Other Online Identifiers, Other (please specify).

Logic

DISABLEMENT LOGIC - Disabled if D01 is no OR if DS06 is yes. OR if DS08 does not contain third parties AND DS10 does not contain third parties.

Scoring Impact

There is no scoring impact associated with this question.

What data is automatically shared with other devices?

ORC_DS16

 Further Information

Guidance/Context

If the policy has clearly stated that data is automatically shared with other devices but is not clear exactly what data is shared then assessors can assume that all collected data is automatically shared. Do not select Biometric data if it is using the phone's native scanner/face ID rather than an in-app one scanner.

Response

Multiple Choice

Answer Criteria

Name (Full Name, Nickname or First Name Only), Username, Mobile Number/Device Number/Home Phone Number, Email, Full Address/Postcode, Age/DOB, Location Data, IP Address, Device IMEI Number, Other Unique Device Identifiers, General Wellness Data, General Identifier e.g. NHS No, Physical Description, Martial Status/Family/Lifestyle/Social Circumstance, Income/Financial/Tax Situation, Education/Qualifications/Professional Training/Awards, Employment/Career History, Cookies/Web Beacons etc. (used for tracking an individual’s online browsing behaviours/movements), Usage Data, Card/Payment/Financial Information, Race/Ethnic Origin, Political/Religious/Philosophical Beliefs or Opinions, Trade Union Membership, Genetic or Biometric Data (where this is used for identification purposes e.g. fingerprint or facial recognition), Physical and/or Mental Health Data, Sexual Orientation or Sex Life, Gender (self-declared or observed), Offences Committed/Alleged to have Committed/Criminal Proceedings/Outcomes/Sentence, Other Online Identifiers, Other (please specify).

Logic

DISABLEMENT LOGIC - Disabled if D01 is no OR if DS06 is yes. OR if DS08 does not contain other devices AND DS10 does not contain other devices.

Scoring Impact

There is no scoring impact associated with this question.

Algorithm/AI

Question

Question Reference Source

Does the app contain algorithms?

ORC_AI01

 Further Information

Guidance/Context

This question aims to identify if there are any algorithms used in the app. This question then influences if other questions around AI and Clinical Calculators are asked later on in the assessment.

Response

Yes / No

Answer Criteria

Yes: If the app uses an algorithm to provide an output, using the health data input by the user OR If the app provides an average from input data, or calories burned.

No:  If the app does not calculate anything with the data it collects OR If the algorithm doesn’t come from health data input.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

How does the app use the algorithm?

ORC_AI02

 Further Information

Guidance/Context

This question allows the assessor to describe what the algorithm does and what area it focuses on. There are questions later on in scene setters which will probe diagnoses/treatment further.

Response

Free Text - E.g. perform a calculation for diagnosis etc.

Answer Criteria

N/A

Logic

DISABLEMENT LOGIC - Disabled if AI01 is no.

Scoring Impact

There is no scoring impact associated with this question.

Does the app appear to use AI?

ORC_AI03

 Further Information

Guidance/Context

This question aims to identify if is any form of AI used within the app. Somethings this may be difficult to decipher from just using the app so the assessor should read around the app to see if the Developer makes these claims.

Response

Yes/ No

Answer Criteria

YES: If the app uses a chatbot which learns from and reacts to what the user says.

YES: If the app/developer claims to use AI techniques.

YES: If the app uses machine learning to improve the quality of its automated decision making.

NO: If the app doesn’t use a chatbot.

NO: If the app/developer makes no claim about using AI techniques.

NO: If the app doesn’t use machine learning to improve the quality of its automated decision making.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

What AI technique is used in the app?

ORC_AI04

 Further Information

Guidance/Context

The Developer may state in app store description or their website what type of AI they use. If it is unclear what AI is used within the app, the assessor should try and find this information through reading the app store description and app website.

Response

Free text

Answer Criteria

Examples: 

Natural Language Processing (NLP) - Includes Natural Language Understanding, Natural Language generation, Machine Translation. E.g. If the app uses a chatbot which learns from and reacts to what the user says.

Machine Learning - If the app uses machine learning to improve the quality of its automated decision making.

Image Recognition - If the app uses AI to identify something in a picture

Logic

DISABLEMENT LOGIC - Disabled if AI03 is no.

Scoring Impact

There is no scoring impact associated with this question.

Is the AI monitored/ maintained?

ORC_AI05

 Further Information

Guidance/Context

To ensure AI is used both appropriately and effectively, humans should have oversight through monitoring/maintaining/updating the app. Developers may monitor their AI by asking healthcare professionals to review the decision making and output. If the output does not appear be in line with the healthcare professionals knowledge, the Developer should correct this. Assessors should look for these mentions via the app, the app/google play store and associated website.

Response

Yes / No

Answer Criteria

YES: If the developer mentions specifically that their AI is monitored/maintained/updated.

No: If there is no specific mention of them updating/maintaining the AI.

No: If they only mention improvements based upon input (learning from input).

Logic

DISABLEMENT LOGIC - Disabled if AI03 is no.

Scoring Impact

There is no scoring impact associated with this question.

Information

Question

Question Reference Source

Is the app designed to provide information or guidance?

ORC_I01

 Further Information

Guidance/Context

This question aims to contribute to the apps functions and features. It is also used to inform the ESF tiering.

Response

Yes / No

Answer Criteria

Yes: If the app provides any generic info around the general topic area, in text or diagram form.

Yes: If the app can provide information as a diary back to the user if monitoring is taking place.

No: If the app provides no real information or guidance aimed at health or wellbeing.

No: If the only information is provided by other users on a forum, information must come from the developer/app itself.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the app provide information that is personalised to an end user’s specific circumstances?

ORC_I02

 Further Information

Guidance/Context

This question aims to contribute to the apps functions and features. It is also used to inform the ESF tiering.

Response

Yes / No

Answer Criteria

Yes: If any of the information provided is personalised to the user. E.g. provides recommended activities/actions based on assessment over a period of time OR tailors therapy/treatment program based on one off assessment which includes taking a lot of information from the user.

No: If the app provides no information which is personalised to the user.

Logic

DISABLEMENT LOGIC - Disabled if I01 is no.

Scoring Impact

There is no scoring impact associated with this question.

Does the app provide users with information regarding where they are able to find local or suitable support services?

ORC_F08

 Further Information

Guidance/Context

This question aims to contribute to the apps functions and features. It is also used to inform the ESF tiering.

Response

Yes / No

Answer Criteria

YES: If the app provides links/signposts to online services or local services.

YES: If the app points to services where the user can take control of their/somebody’s condition e.g. pharmacy.

NO: If the app provides no information which is personalised to the user.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the app provide environmental data not specific to the patient?

ORC_F03

 Further Information

Guidance/Context

This question aims to contribute to the apps functions and features. It is also used to inform the ESF tiering.

Response

Yes / No

Answer Criteria

YES: If the app provides details of external environmental factors which may impact health/wellbeing, such as temperature, pollen count etc.

NO: If the only information provided is the location.

Logic

DISABLEMENT LOGIC - Disabled if I01 is no.

Scoring Impact

There is no scoring impact associated with this question.

Does the App provide information, resources or activities to the public, patients or physicians, either about a specific condition or general health and lifestyle?

ORC_EF07

 Further Information

Guidance/Context

In most cases the answer to this question will be yes because the scope of the question is so broad. The only instance an assessor should answer no to this question is if the app is aimed at providing information, resources or activities for administrative purposes instead of health related purposes. If I01 is yes, this will also be yes. This question also guides the ESF tiering later.

Response

Yes / No

Answer Criteria

Yes: Any app that provides a resource, either condition specific or generalised.

No: Administration apps which have no effect on patient outcomes for instance a schedule system.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Clinical Decision Support - Pre-Diagnosis, Diagnosis and Treatment Support

Question

Question Reference Source

Is the data the app collects, automatically assessed, for the purposes of evaluating: risk; or providing diagnostic support?

ORC_PD01

 Further Information

Guidance/Context

This is looking at apps which provide individual risk to a user, which is personalised based on the user health data collected (e.g. apps which have red zones/percentages of having a condition for specific readings).

Response

Yes / No

Answer Criteria

Yes: If the app provides a calculation, paragraph, table or diagram indicating a users risk, or potential diagnoses.

No: If the app provides no form of risk assessment or diagnoses to a user.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the app diagnose a specific condition?

ORC_DG02

 Further Information

Guidance/Context

This question aims to discover if the app diagnoses/screens/detects a disease or condition (i.e., using sensors, data, or other information from other hardware or software devices, pertaining to a disease or condition). This is a key question for identifying diagnostic medical devices under FDA.

Response

Yes / No

Answer Criteria

Yes: A healthcare professional can see “We think you have..”.

Yes: If the app diagnoses a specified clinical condition using clinical data.

No: If it states ‘you might have a condition please see a professional’, this would be no as it is not specific enough.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the app provide an assessment (of the risk) to an individual - based on data input or collected by the app - of: 

Contracting or Suffering a healthcare condition

The impact on their lifestyle and health indicators 

No Risk Assessment provided

ORC_DG01

 Further Information

Guidance/Context

This is looking at apps which provide individual risk assessments to the user. Both ‘Contracting or suffering a healthcare condition’ and ‘The impact of their lifestyle choices and health and wellbeing indicated’ can be selected at the same time.

Response

Multiple Choice

Answer Criteria

Contracting or suffering a healthcare condition: The app provides a risk indicator of a condition or diagnosis to an individual.

The impact of their lifestyle choices and health and wellbeing indicators: The app provides a risk indicator based on the person's lifestyle to an individual, eg a person’s risk of type 2 diabetes based on food intake.

No risk assessment provided: The app provides no risk indicator or diagnoses.

Logic

DISABLEMENT LOGIC - Disabled if PD01 is no AND DG02 is no.

Scoring Impact

There is no scoring impact associated with this question.

Does the app provide an assessment (of the risk) to a healthcare professional - based on data input or collected by  the app - of: 

Contracting or Suffering a healthcare condition

The impact on their lifestyle and health indicators 

No Risk Assessment provided

ORC_DG03

 Further Information

Guidance/Context

This is looking at apps which provide individual risk assessments to a healthcare professional. Both ‘Contracting or suffering a healthcare condition’ and ‘The impact of their lifestyle choices and health and wellbeing indicated’ can be selected at the same time.

Response

Multiple Choice

Answer Criteria

Contracting or suffering a healthcare condition: The app provides a risk indicator of a condition or diagnosis to an individual.

The impact of their lifestyle choices and health and wellbeing indicators: The app provides a risk indicator based on the person's lifestyle to an individual, eg a person’s risk of type 2 diabetes based on food intake.

No risk assessment provided: The app provides no risk indicator or diagnosis.

Logic

DISABLEMENT LOGIC - Disabled if PD01 is no AND DG02 is no.

Scoring Impact

There is no scoring impact associated with this question.

Does the app provide the option for further assessment or analysis by a healthcare professional?

ORC_DG04

 Further Information

Guidance/Context

This question is applicable to apps which provide individual risk to a user using some kind of algorithm or AI. It aims to identify if the user can send their result/information to a healthcare professional to get a further assessment, for more information.

Response

Yes / No

Answer Criteria

Yes: If the app provides a calculation, paragraph, table or diagram indicating a users risk, and allows this to be sent to a HCP for further investigation/information (being able to get a second opinion from a real clinician by sending information on). 

No: If the app provides no form of further investigation/information by a HCP.

No: If the app offers solely virtual consultations with HCP with no transfer of health data logged within the app.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is the app/does the app include a Symptom Checker?

ORC_DG05

 Further Information

Guidance/Context

This questions aims to identify apps which collect check the user’s symptoms and provide a possible diagnosis/diagnoses based on the inputted information. The purpose or benefit of the app must be its symptom checking functionality. For instance, if an anxiety/depression app contained GAD-7 or PHQ-9, this would not be a symptom checker.

Response

Yes / No

Answer Criteria

Yes: If the app provides a possible diagnosis based upon the collection of a user’s symptoms.

No: If the app provides no form of diagnosis or risk assessment based upon the collection of a user’s symptoms.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the app indicate likelihood of a match for the listed conditions?

ORC_DG06

 Further Information

Guidance/Context

This is looking at whether a symptom checker provides an assessment of what is the most likely cause to the user. For example, 9/10 people who have your symptoms suffer from X.

Response

Yes / No

Answer Criteria

Yes: If the app provides an assessment of chance, or likelihood of certain conditions based on collected symptoms.

No: If the app provides no assessment of chance, or likelihood of certain conditions based on collected symptoms.

Logic

DISABLEMENT LOGIC - Disabled if DG05 is no.

Scoring Impact

There is no scoring impact associated with this question.

Can users filter results to display by highest risk / likelihood / severity?

ORC_DG07

 Further Information

Guidance/Context

This question aims to identify if an app allows the user to filter through the list of conditions the symptom checker provided. It is crucial that the user has the option to turn the filter on or not, if the list is automatically generated in a particular order, this is not providing the user the autonomy to filter. Any filtering rules are applicable from likelihood of symptoms matching a condition to most severe condition symptoms may relate to.

Response

Yes / No

Answer Criteria

Yes: If the app provides a filter for the provided risks. The app needs to provide a specific filter option, and sorting by order of likelihood automatically is NOT sufficient.

No: If the app provides no filter for the provided risks.

Logic

DISABLEMENT LOGIC - Disabled if DG05 is no.

Scoring Impact

There is no scoring impact associated with this question.

Does the app provide treatment recommendations for the listed conditions?

ORC_DG08

 Further Information

Guidance/Context

This question aims to identify if symptom checker apps provide treatment suggestions alongside the listed conditions. If next to a symptom the app recommends a user should seek treatment through signposting to further services, this is not sufficient. The app must be providing the treatment details itself for this question.

Response

Yes / No

Answer Criteria

Yes: If the app provides any treatment suggestions for the listed conditions.

No: If the app provides no treatment options for the listed conditions.

Logic

DISABLEMENT LOGIC - Disabled if DG05 is no.

Scoring Impact

There is no scoring impact associated with this question.

Does the app only signpost the user to suitable care  or recommend seeking further advice? (eg. Go to ER, book an appointment with your family physician, call 911)

ORC_DG09

 Further Information

Guidance/Context

This question aims to identify if the symptom checker app provides suggestions for the user to seek further treatment, based upon the indicated diagnoses. Anything from calling 911 to a recommended visit to your family physician would be sufficient for this question.

Response

Yes / No

Answer Criteria

Yes: If the app provides any signpost to a further service based upon the symptom checker outcome.

No: If the app does not signpost to a further service based upon the symptom checker outcome.

Logic

DISABLEMENT LOGIC - Disabled if DG05 is no.

Scoring Impact

There is no scoring impact associated with this question.

Does the app contain a clinical calculator?

ORC_TS01

 Further Information

Guidance/Context

The FDA takes into account the use of algorithms and AI in relation to identifying and assessing medical devices. The identification of a clinical calculator would help support the argument as to whether an app should be classified as a medical device or not.

Response

Yes / No

Answer Criteria

Yes: This includes apps for use by clinicians or users to calculate parameters pertaining to care, such as early warning system software (if there are treatment implications associated with the calculation). 

No: If the app is not for use by clinicians or users to calculate parameters pertaining to care, such as early warning system software.

Logic

DISABLEMENT LOGIC - Disabled if AI03 is no AND if AI01 is no.

Scoring Impact

There is no scoring impact associated with this question.

What type of clinical calculator does the app contain?

ORC_TS02

 Further Information

Guidance/Context

This question allows the assessor to record information about the type of Clinical Calculator which has been located within the app. For example, contained within the app may be something which calculates the amount of water needed to treat a burns victim.

Response

Free Text

Answer Criteria

N/A

Logic

DISABLEMENT LOGIC - Disabled if TS01 is no.

Scoring Impact

There is no scoring impact associated with this question.

Is the app intended to be (or does the developer claim it can be) used for the prevention of disease?

ORC_MD01

 Further Information

Guidance/Context

Prevention is another key definition in defining a medical device according to FDA regs. If an app is claiming to be used for prevention or its intended use/benefit relates to prevention of a disease or condition it is likely to be a medical device.

Response

Yes / No

Answer Criteria

Yes: If the app is intended to PREVENT a specific disease or condition OR If the actual app will stop you from getting the disease.

No: If the app is trying to catch something early before it develops, e.g. skin vision.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

How does the app prevent disease?

ORC_TS04

 Further Information

Guidance/Context

This question allows the assessor to explain how the app prevents disease.

Response

Free Text

Answer Criteria

N/A

Logic

DISABLEMENT LOGIC - Disabled if MD01 is no.

Scoring Impact

There is no scoring impact associated with this question.

Does the app provide treatment of a condition?

ORC_TS05

 Further Information

Guidance/Context

This question aims to identify if an app provides treatment to a user’s specific condition. This includes both apps that provide information which can be used to enable treatment as well as apps which provide an output which can be used to treat a condition. For example, apps which calculate that are intended to calculate the dose of insulin a diabetic needs to treat their diabetes based on carbohydrate in a meal.

Response

Yes / No

Answer Criteria

Yes: Apps that provide information that can be used to enable treatment to be performed or claim that the output from the app can be used to treat a condition. E.g. an app to calculate the dose of insulin a diabetic needs to treat their diabetes based on carbohydrate in a meal.

No: If the app is intended to treat non-medical conditions e.g. non-specific stress. OR apps intended to just provide tips or advice or link to support groups OR medication reminders.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

What treatment does the app provide?

ORC_TS06

 Further Information

Guidance/Context

This question allows the assessor to explain what treatment the app provides.

Response

Free Text

Answer Criteria

N/A

Logic

DISABLEMENT LOGIC - Disabled if TS05 is no.

Scoring Impact

There is no scoring impact associated with this question.

Does the app guide the treatment of a condition?

ORC_TS07

 Further Information

Guidance/Context

This question aims to identify apps which are guiding the treatment of a condition. This can occur in a number of ways but it is key that the app is guiding the treatment of a condition following best clinical practice guidelines.

Response

Yes / No

Answer Criteria

Yes: Apps which take a user’s health information, and provide specific treatment pathways for the user to follow to treat their condition OR clinician-facing apps that advise treatments.

No: Apps intended to just provide tips or advice which is non-specific to the user.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

How does the app guide the treatment of the condition?

ORC_TS08

 Further Information

Guidance/Context

This question allows the assessor to explain how the app can guide the user’s treatment of a condition.

Response

Free Text

Answer Criteria

N/A

Logic

DISABLEMENT LOGIC - Disabled if TS07 is no.

Scoring Impact

There is no scoring impact associated with this question.

Who does the app provide the treatment guidance to?

ORC_TS09

 Further Information

Guidance/Context

The question allows the assessor to confirm whether treatment guidance is for a general user or a healthcare professional.

Response

Multiple Choice

Answer Criteria

User: Refers to the patient / carer using the app.

HCP: Health Care Professional.

Logic

DISABLEMENT LOGIC - Disabled if TS07 is no.

Scoring Impact

There is no scoring impact associated with this question.

Is the treatment provided independently of a healthcare professional?

ORC_TS10

 Further Information

Guidance/Context

This question aims to determine whether the app can provide treatment to an individual without a healthcare professional involvement.

Response

Yes / No

Answer Criteria

Yes: If the app provides treatment to the user without HCP involvement.

No: If the treatment is not provided independently of a HCP, or if the app provides no treatment.

Logic

DISABLEMENT LOGIC - Disabled if TS05 is no AND if TS07 is no.

Scoring Impact

There is no scoring impact associated with this question.

Does the app support healthcare professionals’ decisions about treatments?

ORC_TS03

 Further Information

Guidance/Context

This question aims to identify apps which supports a decision made by a healthcare professional on a case by case basis. The app must be more than a generic textbook and provide information directed towards healthcare professionals.

Response

Yes / No

Answer Criteria

Yes: The app contributes to a professional’s decision about treatment, so this is for doctors to look at OR Supports decision making on a case by case basis (eg Mersey Burns would be yes (tells clinician how much fluid a patient requires based on percentage burns they have suffered).

No: If the app provides generic, non specific care pathways.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the app follow the path of a procedure/treatment without making any decisions?

ORC_TS12

 Further Information

Guidance/Context

Whilst the FDA regulations do not specify if following a path of a procedure/treatment makes an app a medical device, the assessor will be made aware of the risk which comes with this functionality as well as understanding that additional functionality which could lead the app to be a medical device under FDA regulations.

Response

Yes / No

Answer Criteria

YES:  If the app outlines a treatment / procedure but does not make and communicate any decisions to the user.

NO:  If the app makes any decisions for the user.

Logic

DISABLEMENT LOGIC - Disabled if TS03 is no.

Scoring Impact

There is no scoring impact associated with this question.

Does a healthcare professional make the final decision regarding treatment based on advice and/or options displayed?

ORC_TS13

 Further Information

Guidance/Context

Apps could still potentially be a MD if this is answered as yes, relying on additional functionality too. FDA is unclear about specifics in this area (would depend on other functions too). That said, if this question is answered no, it is very likely the app would be identified as a MD under the FDA.

Response

Yes / No

Answer Criteria

YES: If the app outlines a treatment/ procedure but does not make and communicate any decisions to the user.

NO: If the app makes any decisions for the user.

Logic

DISABLEMENT LOGIC - Disabled if TS03 is no.

Scoring Impact

There is no scoring impact associated with this question.

Does the app automate the treatment pathway for an individual patient?

ORC_TS14

 Further Information

Guidance/ Context

Automating the treatment pathway is a software function that makes the app become a regulated medical device by performing patient-specific analysis and providing patient-specific diagnosis, or treatment recommendations.

Response

Yes / No

Answer Criteria

YES: The app creates the treatment pathway for the user, and does not rely on a HCP.

NO: If the app outlines a set of treatments / procedures but the final decision about which treatment is left to the HCP.

NO: If it is a “one size fits all” pathway that doesn’t take into account individual factors.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is the app intended to be (or does the developer claim it can be) used as a physical intervention to reduce the symptoms or severity of a disease, injury or, physical or mental impairment?

ORC_TS15

 Further Information

Guidance/Context

This question aims to identify if the apps intended purpose is to reduce the symptoms or severity of a disease, injury or physical/mental impairment. The app cannot just support a condition or impairment, the app must provide some sort of functionality to reduce the symptoms. For example, a Tinnitus noise cancelling app reduces the symptoms of Tinnitus. This is to help identify Medical Devices.

Response

Yes / No

Answer Criteria

Yes: If an app that provides a physical output to alleviate the symptoms of an existing condition. For example a Tinnitus noise cancelling app.

No: If an app does not provide a physical output to alleviate the symptoms of an existing condition.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is the app intended to (or does the developer claim it can be used to) compensate an injury or, physical or mental impairment?

ORC_MD07

 Further Information

Guidance/Context

This question aims to identify if an app compensates for a specific injury or physical / mental impairment. It is important that the assessor identifies this is the app's intended purpose and that it is not meant for general use.

Response

Yes / No

Answer Criteria

Yes: Apps which the developer claims can compensate for an injury or handicap or claims that the output from the app can be used for this purpose. For example apps to magnify text specifically for people with visual impairment or apps amplify sounds for people with reduced hearing.

No: If the app provides no link to a specific injury or handicap. For example apps to magnify text but there is no mention of visual impairment in the manufacturer’s claims OR apps to amplify sound  but there is no mention of hearing impairment in the manufacturer’s claims.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the app predict the fertile window?

ORC_CC01

 Further Information

Guidance/Context

This question helps identify if the app’s intended purpose/benefits focuses around conception/contraception. This will in turn help identify if the app is a potential Medical Device as defined by the Food and Drug Administration (FDA).

Response

Yes / No

Answer Criteria

YES: If an app informs the user of when their fertile window is.

NO: If the app does not identify the user’s fertile window.

Logic

DISABLEMENT LOGIC - Disabled if MD08 is no.

Scoring Impact

There is no scoring impact associated with this question.

Does the app claim to be used to prevent pregnancy or to conceive?

ORC_CC02

 Further Information

Guidance/Context

This question helps identify if the app’s intended purpose/benefits focuses around conception/contraception. This will in turn help identify if the app is a potential Medical Device as defined by the Food and Drug Administration (FDA).

Response

Yes / No

Answer Criteria

YES: If an app or related website or description claims to make pregnancies more likely or to be able to prevent pregnancy.

NO: If an app or related website or description does not claim to make pregnancies more likely or to be able to prevent pregnancy.

Logic

DISABLEMENT LOGIC - Disabled if CC01 is no.

Scoring Impact

There is no scoring impact associated with this question.

Does the app use body basal temperature (bbt) recorded through an externally connected thermometer?

ORC_CC03

 Further Information

Guidance/Context

This question helps identify if the app’s intended purpose/benefits focuses around conception/contraception. This will in turn help identify if the app is a potential Medical Device as defined by the Food and Drug Administration (FDA).

Response

Yes / No

Answer Criteria

YES: If an app has an assistive device which can be used to record BBT. Measurements can be input manually after taking a reading.

NO: If an app does not have an assistive device which can be used to record BBT, or it does not record BBT.

Logic

DISABLEMENT LOGIC - Disabled if CC02 is no.

Scoring Impact

There is no scoring impact associated with this question.

Does the app use rhythm, body basal temperature (bbt) and cervical mucus methods to prevent pregnancy or to conceive?

ORC_CC04

 Further Information

Guidance/Context

This question helps identify if the app’s intended purpose/benefits focuses around conception/contraception. This will in turn help identify if the app is a potential Medical Device as defined by the Food and Drug Administration (FDA).

Response

Yes / No

Answer Criteria

YES: If an app uses rhythm/bbt/cervical mucus methods to aid in the prevention of pregnancy or to help conception.

NO: If an app doesn’t use rhythm/bbt/cervical mucus methods to aid in the prevention of pregnancy or to help conception.

Logic

DISABLEMENT LOGIC - Disabled if CC02 is no.

Scoring Impact

There is no scoring impact associated with this question.

Does the developer claim that the app can be used as a natural method of birth control?

ORC_CC05

 Further Information

Guidance/Context

This question helps identify if the app is marketed towards facilitating conception/contraception. This will in turn help identify if the app is a potential Medical Device as defined by the Food and Drug Administration (FDA).

Response

Yes / No

Answer Criteria

Yes: If the app markets itself or claims the user can use the app as a natural method of birth control.

No: If the app does not claim to be a natural method of birth control.

Logic

DISABLEMENT LOGIC - Disabled if CC03 is no AND if CC04 is no.

Scoring Impact

There is no scoring impact associated with this question.

Is the app intended to be used for the control of conception?

ORC_MD06

 Further Information

Guidance/Context

This question aims to identify if apps are to be used to control conception through two or more of the practical methods highlighted in CC01 - CC04. This will in turn help identify if the app is a potential Medical Device as defined by the Food and Drug Administration (FDA).

Response

Yes / No

Assessment Criteria

Yes: If the app appears to be a natural form of contraception AND be intended to be used a way of conceiving based on the above answers (CC01, CC02, CC03, CC04 and CC05).

No: If the app only claims to be a natural form of contraception OR intended to be used as a way of conceiving, but doesn’t claim to be both.

Logic

DISABLEMENT LOGIC - Disabled if CC02 is no.

Scoring Impact

There is no scoring impact associated with this question.

Is the app used in combination with drugs or medication? (e.g. medication reminders)

ORC_AE20

 Further Information

Guidance/Context

This question aims to identify if the app can set medication alerts/reminders, trackers or if the app indicates how much medication the user should take.

Response

Yes / No

Answer Criteria

Yes: If an app provides medication reminders or trackers used as an assistive tool OR if the app influences how much you should take e.g. insulin calculator etc.

No: If an app does not provide medication reminders or trackers used as an assistive tool OR if there are no alarms to take medications.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is the app a companion of the device, as opposed to having been designed to connect with a third party manufacturer's device?

ORC_F26

 Further Information

Guidance/Context

This question aims to understand the context of the app and to also examine whether or not it can be used without the device.

Response

Yes / No

Answer Criteria

Yes: The app is designed to work with a specific device, and likely doesn’t function fully without it e.g. Garmin Watch with Garmin App.

No: The app connects with third party devices e.g. Fitbit watch.

Logic

DISABLEMENT LOGIC - Disabled if DC01 does not contain Devices.

Scoring Impact

There is no scoring impact associated with this question.

Monitoring

Question

Question Reference Source

Does the app allow the monitoring of key health information?

ORC_MN01

 Further Information

Guidance/Context

Due to a logic issue, only one answer should be selected. If the app allows the monitoring of both General Health or Wellness and Specific Condition Data, the assessor should select Specific Condition data. This question will contribute to the outcome of MN04, a question which helps determine what ESF tier the app belongs in.

Response

Multiple Choice

Answer Criteria

Yes - General health or Wellness data: If the app has any capability at all which allows the user to monitor any health information which the app records.

Yes - Specific Condition data: If the app is aimed towards someone with a pre-existing condition e.g. chronic pain, diabetes etc.

No - None: If an app does not collect or allow the monitoring of health information.

Logic

DISABLEMENT LOGIC - Disabled if D01 is no. OR disabled if DT10 does not contain neither Physical and/or Mental Health Data nor General Wellness Data.

Scoring Impact

There is no scoring impact associated with this question.

Does the app involve the recording of relevant data over time for the user to access and review (with no ‘intelligent’ manipulation of that data by the app)?

ORC_MN02

 Further Information

Guidance/Context

This question aims to identify if the app allows users to record health information which can be reviewed at a later date. This question contributes to the outcome of MN04, a question which helps determine what ESF tier the app belongs in.

Response

Yes / No

Answer Criteria

Yes: If an app allows the recording and reviewing of data over a period of time to allow the user to monitor their health information.

No: If an app does not collect or allow the monitoring of health information.

Logic

DISABLEMENT LOGIC - Disabled if MN01 contains None.

Scoring Impact

There is no scoring impact associated with this question.

Does the app involve the automated assessment or interpretation of relevant data to deliver alerts, insights, reminders or adjustments regarding the user’s health or lifestyle?

ORC_MN03

 Further Information

Guidance/Context

This questions aims to identify is an app provides further insight around the user’s health data it collects. The app needs to be providing novel insights, automated alerts or reminders from the user’s health data. If the health data is relayed back to the user with no additional information, MN03 will be no. This question contributes to the outcome of MN04, a question which helps determine what ESF tier the app belongs in.

Response

Yes / No

Answer Criteria

Yes: If an app allows the user to record health data, and then the app provides insight back to the user. For example in the form of alerts, reminders or adjustments regarding the user’s health/lifestyle.

No: If an app doesn’t collect health data, or if it collects it and regurgitates it back to the user in the form of a simple graph, without any further insight or information.

Logic

DISABLEMENT LOGIC - Disabled if MN01 contains None.

Scoring Impact

There is no scoring impact associated with this question.

Is the app:

ORC_MN04

 Further Information

Guidance/Context

This questions aims to differentiate between the different types of self-management and therefore different tiers from the Evidence Standards Framework. MN01, MN02 and MN03 all feed into the outcome of this question. Below is a diagram assessors refer to during the assessment process in order to decipher what type of self management tool they reviewing.

Response

Multiple Option

Answer Criteria

A Simple Self Management app: If an app is simple monitoring with wellbeing and general health focus = Tier Bi

A Standard Self Management app: If an app is simple monitoring with a specific condition focus OR complex monitoring with a wellbeing and general health focus = Tier Bii

A Complex Self Management app: If an app is complex monitoring with a specific condition focus = Tier C

Logic

DISABLEMENT LOGIC - Disabled if MN01 contains None.

Scoring Impact

There is no scoring impact associated with this question.

Is the output of the app’s monitoring intended to affect the treatment of an individual?

ORC_MN05

 Further Information

Guidance/Context

This questions aims to evidence whether the monitoring the app performs is specifically there to impact on the user’s treatment.

Response

Yes / No

Answer Criteria

Yes: If the app provides a calculated output which is based on the health information collected, which may directly impact an individual’s decision regarding the treatment management of a condition. For example, a peak flow meter which shows decreasing measurements which acts as an early warning software.

No: If the app is not intended to affect the treatment management and is only carrying out complex monitoring that may display trends or other interesting data points.

Logic

DISABLEMENT LOGIC - Disabled if MN04 does not contain Complex Self Management app.

Scoring Impact

There is no scoring impact associated with this question.

Does the app allow others (i.e. not the user) to monitor or view the health data captured?

ORC_MN06

 Further Information

This question aims to evidence whether the app allows the monitoring of the user’s health data by people who are not the user.

Response

Yes / No

Answer Criteria

Yes: If the app provides functionality within it that allows someone to monitor the user’s collected health data. This may be a HCP or may be a family member or friend.

No: No functionality within the App for someone who is not the user to view the data collected within the app. Functionality needs to be within the app.

Scoring

DISABLEMENT LOGIC - Disabled if MN01 contains None.

Logic

There is no scoring impact associated with this question.

Does the app automatically measure and/or record data about a user’s specified condition, and transmit the data to a professional, caregiver or third party organisation, without any input from the user?

ORC_MN07

 Further Information

Guidance/Context

This question aims to identify whether the app automatically collects and sends health data, without any sort of intervention from the user.

Response

Yes / No

Answer Criteria

Yes: If the app collects user health data and transmits it to somebody else, without any sort of user intervention.

No: The app does not automatically collect data OR the app does not automatically transmit data.

Logic

DISABLEMENT LOGIC - Disabled if MN07 does not contain Specific Condition data.

Scoring Impact

There is no scoring impact associated with this question.

Does the app generate any alarms or alerts from the data recorded by the app or a connected device?

ORC_MN08

 Further Information

Guidance/Context

This question aims to look at whether the app sends a notification to the user/carer/healthcare professional based on any of the data recorded through the app itself or a connected device. For example, a diabetes app could automatically alert the user by creating a noise that notifies the user their blood glucose levels are either hypo (low) or hyper (high) in regards to the satisfactory levels they should usually be.

Response

Yes / No

Answer Criteria

Yes: If the app sends an alarm, alert or notification based on any of the data collected by the app itself or a connected device.

No: The app does not generate alarms based on the health data input and the user has to set them themselves.

Logic

DISABLEMENT LOGIC - Disabled if MN06 is no.

Scoring Impact

There is no scoring impact associated with this question.

Are the alarms generated by user-defined filtering rules?

ORC_MN09

 Further Information

Guidance/Context

This question aims to identify if the user can define the filtering rules surrounding the health data and choose what boundaries trigger an alarm / alert.

Response

Yes / No

Answer Criteria

Yes: If the app alerts the user or HCP to a predefined abnormality manually set by the user.

No: The app does not generate alarms based on the health data input.

No: The app does generate alarms based on health data input but the user can not set these parameters themselves.

Logic

DISABLEMENT LOGIC - Disabled if MN08 is no.

Scoring Impact

There is no scoring impact associated with this question.

What type of intervention or treatment does the app provide?

ORC_TS11

 Further Information

Guidance/Context

This is to determine what type of treatment the app supplies. Assessors can select more than one, for example if an app had a diary and an insulin calculator both Self-management and Monitoring should be selected. This question helps the assessor place the app in the correct ESF tier.

Response

Multiple Choice

Answer Criteria

Preventative behaviour change: If the app is intended to modify the users behaviour to reduce the risk of a condition.

Psychological intervention: If the app is intended to provide a psychological intervention to someone with a diagnosed psychological condition e.g. not non-specific stress.

CBT: If the app is intended to provide Cognitive Behavioural Therapy to a user in full.

Fertility: If the app is intended to be used to help with fertility treatments.

Self-management (administering measures): If the app is intended to be used to help provide information about how much medicine should be taken e.g. diabetic patient advised to take X units of insulin based on information inputted into the app.

Tailored treatment plan: If the app provides the user with a tailored treatment plan to improve their condition based on collected information.

Monitoring (basic): E.g. diary.

Logic

DISABLEMENT LOGIC - Disabled if TS05 is no AND if TS07 is no AND MN01 is None.

Scoring Impact

There is no scoring impact associated with this question.

Online Consultations

Question

Question Reference Source

Can the app be used for patients to have online consultations, conversations, or related Health Care services with a healthcare professional?

ORC_F14

 Further Information

Guidance/Response

This question provides further context around the functionality of the app.

Response

Yes / No

Answer Criteria

Yes: If the app allows the user to access consultation with relevant professionals. This would be a call or online chat directly with a doctor or professional in the relevant field.

No: The app does not allow consultations or other communication with a relevant professional through the app.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is this through video consultation?

ORC_OC02

 Further Information

Guidance/Context

This question follows on from the previous questions and it is also a data capture question. It aims to determine how exactly online conversations and consultations are held.

Response

Yes / No

Answer Criteria

Yes: If the user can have a video consultation directly with a relevant profession via the app.

No: If a video call consultation with the relevant professional is not available via the app.

Logic

DISABLEMENT LOGIC - Disabled if F14 is no.

Scoring Impact

There is no scoring impact associated with this question.

Does the app allow healthcare professionals to provide clinical advice, as opposed to the app providing advice itself?

ORC_EF09

 Further Information

Guidance/Context

This question provides further context around the functionality of the app and importantly informs the user where the advice comes from.

Response

Yes / No

Answer Criteria

YES: If the app enables a HCP to provide advice in whatever format through the app. This may be video consultation, instant messaging or other platform communications.

NO: The app does not allow consultations or other communication from a relevant professional through the app.

Logic

DISABLEMENT LOGIC - Disabled if F14 is no.

Scoring Impact

There is no scoring impact associated with this question.

If the app allows healthcare professionals to provide clinical advice through the app, rather than the app providing the advice itself, how does it do this?

ORC_OC01

 Further Information

Guidance/Context

This question is a data capture question. It aims to collect information about exactly how the app allows a professional to supply clinical advice.

Response

Free text

Answer Criteria

N/A

Logic

DISABLEMENT LOGIC - Disabled if EF09 is no.

Scoring Impact

There is no scoring impact associated with this question.

Administrative Services

Pharmacy

Question

Question Reference Source

Does the app allow users to order and request prescriptions?

ORC_F13

 Further Information

Guidance/Context

This question helps identify if an app can help a user order / request prescriptions.

Response

Yes / No

Answer Criteria

Yes: If the app allows users to order or request a prescription from a healthcare professional, healthcare provider or pharmacy.

No: If the app allows the user to record what prescription they would like to request. This would only be acting as a reminder to the user.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Reminders/Notifications

Question

Question Reference Source

Does the app send push notifications?

ORC_D29

 Further Information

Guidance/Context

This is an information capture question which helps inform the functions and features.

Response

Yes / No

Answer Criteria

Yes: If the app sends push notifications to the device.

No: If there are in-app notifications which are not pushed to the device.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the app send email notifications?

ORC_D30

 Further Information

Guidance/Context

This is an information capture question which helps inform the functions and features.

Response

Yes / No

Answer Criteria

Yes: If the app sends email notifications relating to the user’s use of the app, personalised.

No: If the only emails are marketing/newsletters.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

External Device

Question

Question Reference Source

Is the app's main functionality dependent on the user having one of the devices to connect with the app?

ORC_F27

 Further Information

Guidance/Context

This question aims to identify if an app can only be used for its intended purposes if a user has access to one of the connected devices.

Response

Yes / No

Answer Criteria

Yes: If the app ONLY works with a companion device. For instance, the user is unable to input data and therefore cannot use the app at all without the device.

No: If there is a companion device but the app can still be used independently.

Logic

DISABLEMENT LOGIC - Disabled if DC01 does not contain Devices.

Scoring Impact

There is no scoring impact associated with this question.

Do any of the features or functions of the app appear to allow it to be used to control a medical device?

ORC_F30

 Further Information

Guidance/Context

This question aims to

Response

Yes / No

Answer Criteria

Yes: If the app is used to control an external medical device.

No: The app connects with an external device which is not classified as a medical device.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Forums and Contacts

Question

Question Reference Source

Are there opportunities to link with other users (buddying, forums or group education)?

ORC_U19

 Further Information

Guidance/Context

This is an information capture question which helps inform the functions and features.

Response

Yes / No

Answer Criteria

Yes: If there is any way for users to communicate with other users within the app. This can be through messaging, internal forums, connecting with friends, communicating with a healthcare professional etc.

No: If you can only send a report to a doctor via email for example.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the app provide an internally hosted forum or online community for their users?

ORC_FC01

 Further Information

Guidance/Context

This question refers to forums which are within the app rather than ones hosted externally via Facebook, developer website etc.

Response

Yes / No

Answer Criteria

Yes: If the app has an internal forum.

No: If the app provides links to a third party forum or an externally hosted forum. One-to-one communication is not a forum.

Logic

DISABLEMENT LOGIC - Disabled if U19 is no.

Scoring Impact

There is no scoring impact associated with this question.

Does the app link to a third-party service to host a forum or online community for their users?

ORC_FC02

 Further Information

Guidance/Context

This question refers to forums which are hosted eternally via Facebook, developer website etc. rather than within the app.

Response

Yes / No

Answer Criteria

Yes: If the app provides links to a third party forum or an externally hosted forum.

No: If the app links to a Facebook page which is not a forum. If the only forum is in-app.

Logic

DISABLEMENT LOGIC - Disabled if U19 is no.

Scoring Impact

There is no scoring impact associated with this question.

Does the app allow two-way communication between citisens, patients or healthcare professionals?

ORC_EF10

 Further Information

Guidance/Context

This question is an information capture question which informs functions and features. The two way communication needs to exist within the app through chat functions, a forum, video call and must be between two or more people.

Response

Yes / No

Answer Criteria

Yes: If the app allows for any two-way communication between any two people.

No: If the app does not enable two-way communication between two or more people.

Logic

DISABLEMENT LOGIC - Disabled if U19 is no.

Scoring Impact

There is no scoring impact associated with this question.

Goal Setting

Question

Question Reference Source

Does the app provide gamification or goal setting features for the user?

ORC_F06

 Further Information

Guidance/Context

This question is an information capture question which informs functions and features. The gamification or goal setting features must somehow relate to the user’s health or wellbeing.

Response

Yes / No

Answer Criteria

Yes: If you can choose a goal, get badges or achievements through use of the app.

Yes: If the app provides targets or you can set your own targets.

Yes: If the app encourages engagement with rewards.

No: No goal setting or gamification. If gamification has no real purpose.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the app set goals for the user?

ORC_GS01

 Further Information

Guidance/Context

This question follows on from the previous and aims to identify what type of goals exist in the app.

Response

Multiple Choice

Answer Criteria

Tailored: If the goals are specific to the user. For example, the user can input health parameters and the app generates a goal based on those readings. 

Generic: If the set goals are generic for all users. For instance, goals that are pre-set within the app and are the same for each user. 

User defined: If the user can manually or directly specify or customise their goal. For example, the user can choose a weight loss goal which they can set themselves. 

Logic

DISABLEMENT LOGIC - Disabled if F06 is no.

Scoring Impact

There is no scoring impact associated with this question.

Does the app allow the user to set goals for themselves?

ORC_U21

 Further Information

Guidance/Context

This question allows us to understand further the functionality of the app, it helps us with our adapted ESF tiering.

Response

Yes / No

Answer Criteria

Yes: If ‘User defined' has been selected in the previous question (ORC_F06).

No: If only ‘Tailored’ or ‘Generic’ has been selected in the previous question (ORC_F06).

Logic

DISABLEMENT LOGIC - Disabled if F06 is no.

Scoring Impact

There is no scoring impact associated with this question.

Customisation

Question

Question Reference Source

Can the app presentation be customised by the user?

ORC_CUS01

 Further Information

Guidance/Context

This question aims to identify if the user can edit the style of the app to suit their needs and / or preferences.

Response

Yes / No

Answer Criteria

Yes: If any changes can be made to the presentation theme within the app. This includes editing the background, colours, profile picture, language, measuring units etc.

No: If the presentation of the app cannot be edited or customised by the user in any way.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the app respond to preferences in the device?

ORC_CUS02

 Further Information

Guidance/Context

This question aims to identify if the user can set preferences on the device which is carried through to the app to suit their needs and / or preferences.

Response

Yes / No

Answer Criteria

Yes: If the app responds to changes in font size.

Yes: If the app provides support options for users with poor vision/poor hearing.

No: If the app only responds to in-app preferences.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Business Model

Question

Question Reference Source

Is the app totally free?

ORC_U29

 Further Information

Guidance/Context

This is an information capture question which used to inform the users whether the app is completely free or if there are some sort of costs involved in the app. If the app has any costs associated with it ranging from in app purchases to licenses required by a healthcare provider, the assessor should answer this question no. If the app requires the user to purchase the associated device in order to use the app, the assessor should answer this question no.

Response

Yes / No

Answer Criteria

Yes: If the app is free to download AND has no in-app purchases or subscriptions AND costs are not incurred or covered by any third party organisation/employees. For instance, licenses are NOT needed to be purchased for distribution).

No: If licenses are needed to be purchased for distribution.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

How is the app funded?

ORC_BM01

 Further Information

Guidance/Context

This question aims to identify what the business model is behind the app. If the answer is not apparent through publicly available information, the assessor should select Self-funded as a default.

Response

Multiple Choice

Answer Criteria

In-app purchase: If the app is funded through the user purchasing something within the app after downloading.

Subscription: If the app is funded through subscription fees which the user has to pay in order to download the app.

One off payment: If the app is funded through one off payments which the user needs pay in order to download the app.

Licensed by doctor/healthcare provider: If the app is funded through licenses which need to be purchased in order for doctors/healthcare providers to provide access to their patients.

Donations: If the app is funded through donations.

Government or similar grant: If the app is funded through a government or similar grant.

Charity / Non profit: If the app is funded by a charity or non-profit organisation.

Self-funded: If the app is self-funded by the people who run the company. OR if there is no evidence of how the app is funded then Self funded should be selected.

Logic

DISABLEMENT LOGIC - Disabled if U29 is no.

Scoring Impact

There is no scoring impact associated with this question.

Does the app contain advertisements?

ORC_U27

 Further Information

Guidance/Context

This question aims to identify if the app displays advertisements for external products/services. If the app advertises their own subscriptions or in app purchases, this does not count.

Response

Yes / No

Answer Criteria

Yes: If the app has adverts for other products/services within it.

No: If the app contains adverts for its own in-app purchases.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Benefits

Question

Question Reference Source

What are the claimed or implied benefits of the App?

ORC_BF01

 Further Information

Guidance/Context

This questions aims to identify the intended purpose of the app through highlighting the claimed / implied benefits. If the assessor reads a clear benefit described by the developer, this would be a claimed benefit. If the assessor is having to infer a benefit from text written by the developer, this would be an implied benefit. If a claimed or implied benefit does not appear in the list below, the assessor should select Other Claimed/Implied Benefit and make the benefit very clear. In order for evidence to meet the requirements of the framework, the evidence of efficacy should relate to the benefits / intended purposes of the app.

Response

Multiple Choice

Answer Criteria

Cost savings to the healthcare system

Increased access to care

Improved diagnostic or risk assessment

Improved quality of treatment

Improved recovery

Reduced readmission or re-referral

Improved management of a condition

Preventative Behaviour Change

Improved mental wellbeing

Improved physical wellbeing

Improved system/process efficiency

Other Claimed Benefit (please describe)

Other Implied Benefit (please describe)

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

DATA & PRIVACY

Privacy Policy

Initially, the evaluation identifies the relevant privacy policy for the app, which is available to users through the app and/or the App Store or Play Store and/or on the website. The more transparent the privacy policy, the better. Ultimately, the privacy policy must clearly state that user data will not be used or shared with other parties, except as described in the privacy policy, or without express consent of the user. Ideally it will identify:

·         what data is collected from the user and how,

·         if the user is informed of the developer’s intentions with processing and sharing their data, and

·         if the user’s consent is obtained.

The privacy policy should accurately reflect the data usage of the app. The Assessors will be able to note if any data is collected outside of what is detailed in the privacy policy. Additionally, the policy should inform users of the developer’s intent to use their data for marketing purposes. If user data is shared for any other purposes other than basic use of the app, or legal obligations, then the evaluation considers if the user is able to opt out of these activities.

Privacy Policy

Question

Question Reference Source

Is there a privacy policy clearly available via the Web App/ Website? 

ORC_D39a

 Further Information

Guidance/Context

This question should only be active if personal and/or sensitive data is collected by the app and is shared with the developer/third parties etc. If this appears to be wrongly active then check back on data scene setters.

Response Type

Yes/No

Answer Criteria

Yes: If any data is collected by or through the app, in any way. Including data such as usage data, cookies etc.
No:

Logic

DISABLED LOGIC -This question should only be active if personal and/or sensitive data is collected by the app and is shared with the developer/third parties etc. If this appears to be wrongly active then check back on data scene setters.

Scoring Impact
Maximum risk applied to the data section in this question and all questions that are disabled as a result of answering D39a as No. Questions would be disabled as a result of them not being applicable due to the app not having an applicable policy.

Is there a privacy summary published anywhere by the developer?

ORC_D39b

 Further Information

Guidance/Context

Due to the nature of the data being collected being non-identifiable a summary privacy is suitable. This question should only be active if personal and/or sensitive data is not collected by the app or is not shared with the developer/third parties etc. If this appears to be wrongly active then check back on data scene setters.

Response Type

Yes/No

Answer Criteria

Yes: A privacy summary can be a simple paragraph explaining privacy practices of the developer, as collection of non-personal/sensitive data does not require a full privacy policy.

Logic

DISABLED LOGIC -Disabled if data is automatically shared/collected. Only enabled when the app/developer collects only none sensitive data OR when the personal/sensitive data is only shared through direct manual intervention from the user.

Scoring Impact

Maximum risk applied to this question and all questions that are disabled as a result of answering D39b as No. Questions would be disabled as a result of them not being applicable due to the app not having an applicable policy.

Is the privacy policy made immediately available when the user first opens the app?

ORC_DP03

 Further Information

Guidance/Context

This question looks to identify when a privacy policy is presented to a user. Making users aware of how their data is collected and used is mandatory.

Response Type

Yes/No

Answer Criteria

Yes: If the privacy policy is displayed when the app is first opened.

Yes: If the user is prompted to view and/or provided with a link to the policy when the app is first opened or on the login page.

Logic

DISABLED LOGIC -Disabled if D39a AND D39b are answered no.

Scoring Impact

High value applied if Yes. Value cannot be applied for both DP03 and DP04.

Is the privacy policy made available when the user is signing up to the service?

ORC_DP04

 Further Information

Guidance/Context

This question looks to identify when a privacy policy is presented to a user. Making users aware of how their data is collected and used is mandatory.

Response Type

Yes/No

Answer Criteria

Yes: When the user is provided with the privacy policy during the sign up process.

No: If the user is not provided with, or linked to the privacy policy during sign up.

Logic

DISABLED LOGIC - Disabled if D39a AND D39b are no, OR if DT14 is no.

Scoring Impact

High value applied if Yes. Value cannot be applied for both DP03 and DP04.

Is it published within the app? 

ORC_DP01

 Further Information

Guidance/Context

A privacy policy must be accessible to the user. This and following question look to identify where the privacy policy is located. Publishing within the app or being accessible via the app result in high value than it only being identifiable on the relevant app store.

Response Type

Yes/No

Answer Criteria

Yes: If the privacy policy is readily available to read at any time within the app.

No: If the privacy policy link takes you out of the app to a web browser.

Logic

DISABLED LOGIC - Disabled if D39a AND D39b are answered no.

Scoring Impact

Medium value applied if Yes. Value cannot be applied for both DP01 and DP02 if both are answered Yes.

Is it available externally via the app, or via a linked website?

ORC_DP02

 Further Information

Guidance/Context

A privacy policy must be accessible to the user. To determine if a policy is external, a user can enter the app manager screen. If still on the app the policy is internal, if an internet browser has opened separate from the the app then it is external.

Response Type

Yes/No

Answer Criteria

Yes: If the policy links outside of the app to the browser.

Yes: If there is an external link to the website, where there is access the privacy policy. This comes under the 2 click rule. Meaning that a privacy policy is easily accessible within 2 clicks/taps

Logic

DISABLED LOGIC - Disabled if D39a AND D39b are answered no.

Scoring Impact

Medium value applied if Yes. Value cannot be applied for both DP01 and DP02 if both are answered Yes.

Is it available via the relevant app store?

ORC_DP05

 Further Information

Guidance/Context

A privacy policy must be accessible to the user. This and following question look to identify where the privacy policy is located. Publishing within the app or being accessible via the app result in high value than it only being identifiable on the relevant app store.

Response Type

Yes/No

Answer Criteria

Yes: If the policy is accessible through the app store, making sure the privacy policy applies to the app. If it doesn’t link directly make sure it is accessible within 2 clicks.

Logic

DISABLED LOGIC - Disabled if D39a AND D39b are answered no.

Scoring Impact

Low value applied if Yes.

What data does the privacy policy state the developer collects? 

ORC_DP06

 Further Information

Guidance/Context

This is a multiple choice question. Choices should be selected based on what is stated in the privacy policy. This would normally be found in a section titled like “What Information do we collect?”. Choices should only be selected if the privacy policy states them, this question should not be based off what can be seen in the app.

Response Type

Multiple Choice

Answer Criteria

Sensitive - Physical / Mental Health or Condition (past, current or future status), Sexual Life / Orientation, Political, Religious or other beliefs or opinions, Offences Committed / Alleged to have Committed / Criminal Proceedings / Outcomes / Sentence,  Financial data (that might be used for payment fraud), Trade Union membership, Racial / Ethnic Origin, Genetic or Biometric Data (e.g. fingerprints / facial Recognition) for the purpose of uniquely identifying a person

Personal (combined - If a number of these items have been selected, then there is a possibility that data can be personally identifiable) - Cookies, web beacons, flash cookies, server logs etc which track individual’s browsing behaviour, Other Unique Device Identifiers eg. Device MAC Address, Name, Age/DOB, Gender (self declared or observed), Marital Status |Family / Lifestyle / Social Circumstance, Education / Qualifications / Professional Training / Awards, Other online identifiers / Event Logs, Location Data (Travel / GPS / GSM Data / radio frequency identification tags (RFID))

Personal - Address|Postcode (full), Email Address, Mobile Phone Number / Device Number / Home Phone Number, Physical Description, Username, IP Address, General Identifier e.g. NHS No, Income / Financial / Tax Situation, Employment / Career History, Device IMEI No

Non-Personal - General Wellness data

Logic

DISABLED LOGIC - Disabled if D39a AND D39b are answered no.

Scoring Impact

There is no scoring impact for this question.

Is the policy accurate, with regards to the data the developer intends to collect?

ORC_DP07

 Further Information

Guidance/Context

This questions looks to capture if the data that is stated as collected by the developer within the privacy policy matches what has been identified during assessment and usage of the app.

Response Type

Yes/No

Answer Criteria

Yes if DP06 contains the same selections as DT10.

Logic

DISABLED LOGIC - Disabled if D39a AND D39b are answered no.

Scoring Impact

High risk applied if No AND DT10 and DP06 do not match.

Does the app state that data collected by the app is stored locally, unless the user manually exports the data?

ORC_D10a

 Further Information

Guidance/Context

This question looks at whether data is stored within the app. An app that requires data to be automatically transferred off the app - even to just be stored remotely - would not meet this requirement.

Response Type

Yes/No

Answer Criteria

Yes: “Stored locally on device” is clearly stated.

Yes: Data is stored only on the device, unless a user chooses to share it, or no data is collected or stored by the developer. 

No: Doesn’t state that personal data is stored only on the device.

No: Personal data is clearly transferred to and stored in any location outside the device with no involvement from the user.

Logic

DISABLED LOGIC - Disabled if D01 OR DS06 are No, or if DS07 OR DS09 are answered Yes.

Scoring Impact

High value if Yes.

How does the developer obtain consent for the processing of user data?

ORC_DP08

 Further Information

Guidance/Context

Consent should be obvious and require a clear, positive, physical action from the user to opt in. Consent requests must be prominent, separate from other terms and conditions, easy to understand, and user friendly.

During sign-up to the app attention should be paid to how, if at all, consent is obtained from the user.

Response Type

Multiple Choice

Answer Criteria

Unmarked opt in check box, separate from other terms and conditions and/or consent requests (separate boxes for privacy policy, terms/conditions and marketing).- if there is an unmarked checkbox where the user can agree or consent to the privacy policy alone.

Clear affirmative acceptance option, separate from acceptance of other terms and conditions and/or consent requests (separate acceptance option for privacy policy, terms/conditions and marketing).- if there is another form of acceptance of the privacy policy, eg. clicking “sign up” after having been presented with the privacy policy.

Explicitly through express confirmation in words, rather than any other positive action (e.g. the user is required to email/write to the developer providing a clear confirmation of consent). This does not apply to a statement in the privacy policy such as “by using this app you consent to us collecting your data.”)- if the user is required to email the developer to provide their written consent.

Another form of positive action to opt in to giving consent (please detail below) - eg. if the acceptance box is for both privacy policy and T&Cs.- if there is an unmarked checkbox to agree to the privacy policy and T&Cs all together.

Other (please detail below), e.g. A statement in the privacy policy such as by using this app you consent to us collecting your data, with no clear confirmation of acceptance of policy. - if there is no clear option to be taken by the user to accept the privacy policy.

Logic

DISABLED LOGIC - Disabled if D39a is No.

Scoring Impact

Very high risk applied if “Other” is selected + multiplier based on nature of the data.

Does the privacy policy provide the name and contact details of their Data Protection Officer (DPO), or similar individual representative for the company?

ORC_DP14

 Further Information

Guidance/Context

A DPO is important to ensure, in an independent manner, that an organisation applies the laws protecting individuals' personal data.

Response Type

Yes/No

Answer Criteria

Yes: If an individual person has been named and declared the person responsible for the company’s privacy practices, with contact details (this can include a generic email, such as dpo@company.co.uk, providing the individual responsible has been named).

No: If an individual person who is responsible for the role of DPO has not been named/detailed.

No: If there is only a generic email address.

Logic

DISABLED LOGIC - Disabled if D39a is no.

Scoring Impact

High value applied if yes.

Provide the details of the DPO: (Text Response)

ORC_DP15

 Further Information

Guidance/Context

Input the details of DPO from within privacy policy, this should be a named person not just a generic email and “data protection officer”.

Response Type

Free Text

Answer Criteria

N/A

Logic

DISABLED LOGIC - Disabled if DP14 has not been answered yes.

Scoring Impact

None

 

Data use

Once it is established what data is collected by the app, the evaluation looks at how that data is used and shared, and if this is communicated to the user. The privacy policy should state all intended uses and legal basis of processing user data, such as legal obligation, research or marketing. Users should also be given the option to withdraw consent for the use of their data, particularly for marketing.

Question

Question Reference Source

Does the developer fully inform the user of how they will collect data about them?

ORC_D69

 Further Information

Guidance/Context

This questions identifies if the developers has clearly stated in the privacy policy how data will be collected from users. For example “ data will be collected when registering to use the app”.

Response Type

Yes/No

Answer Criteria

Yes: If the developer informs users where any data will be collected from. Eg. directly from the user or through third party sources.

No: If the developer has not informed users of all potential sources of information about them. Eg. the user is informed of data collected about them, however, the developers fail to identify that information is obtained from another location, such as Facebook, when the user signs up with this account.

Logic

DISABLED LOGIC - Disabled if D39a and D39b have not been answered yes OR DS06 has been answered Yes.

Scoring Impact

Medium risk applied if No + multiplier based on the nature of the data..

Does the developer provide users with details on all the purposes of processing user data?

ORC_D13

 Further Information

Guidance/Context

This question looks to identify if the purpose of processing has been made clear. For example, a developer may state that email addresses are captured to share marketing information with users.

Response Type

Yes/No

Answer Criteria

Yes: If the policy clearly explains what the user data collected is used for.

Yes: If the policy states all the uses for collected data that are apparent from the app.

No: If there is reason to believe that the developer has not explained any of the purposes for processing user data (Please detail in comments section).

Logic

DISABLED LOGIC - Disabled if D39a and D39b have not been answered yes OR DS06 has been answered yes.

Scoring Impact

Medium risk applied if No + multiplier based on the nature of the data.

What is automatically shared data used for?

ORC_DP10

 Further Information

Guidance/Context

Selection of answers for this questions should apply to data automatically shared with third parties/HCP/other users/devices - NOT with the developer. The exception to this is marketing should be selected if this a purpose of data sharing with the developer.

Response Type

Multiple Options

Answer Criteria

Legal obligations,

Performance of contract,

Payment transactions,

Research,

Improving of developer services,

Marketing,

Provision of service,

Other (Please specify),

Unclear.

Logic

DISABLED LOGIC - Disabled if D39a and D39b have not been answered yes OR DS06 has been answered Yes.

Scoring Impact

None

Does the developer appear to intend to share or process the user data collected by the app for any purposes that have not been made clear to the user, or for any purposes they deem necessary?

ORC_D38

 Further Information

Guidance/Context

This question is asking if there is the possibility that data is being shared without this being made clear to the user. Therefore, No is the positive response.

Response Type

Yes/No

Answer Criteria

Yes: If data is shared without user consent, AND users don’t need to agree to the privacy policy. Essentially the opposite of D16.

Yes: If there is an obvious purpose for data use, which isn’t made clear or mentioned in the privacy policy. 

Yes: The policy states that the data will not be shared without first obtaining the user's consent to do so or that the app/developer ‘Won't share for other reasons/ with other parties, except as has been set out in the policy without obtaining your consent’

Logic

DISABLED LOGIC - Disabled if D39a has not been answered Yes.

Scoring Impact

High risk applied if yes + multiplier based on the nature of the data.

Does the developer inform users that they would like to use their data for the purpose of marketing?  

ORC_D71

 Further Information

Guidance/Context

If direct marketing is being undertaken then developers need separate additional consent from the users. Answers for this questions are typically found in a section with name similar to “What we do with the information we collect” or the developers may have a separate “Marketing” section.

Response Type

Yes/No

Answer Criteria

N/A

Logic

DISABLED LOGIC - Disabled if D39a has not been answered yes or DT13 is answered as non-personal or D01 is answered no.

Scoring Impact

None

Does the developer obtain informed consent separately, for the purpose of marketing?

ORC_DP12

 Further Information

Guidance/Context

Consent for marketing should be obtained separately from consent for any processing user data for any other purpose. This must also be prominent, easy to understand and user friendly. E.g. A separate tick box for marketing and consenting with the privacy policy.

Response Type

Yes/No

Answer Criteria

Yes: If consent for marketing is obtained separately for marketing AND the method for gaining this consent is through one of the positive affirmative actions listed in DP08. (Unmarked opt in check box; clear affirmative acceptance option; explicitly through express confirmation in words, another form of positive action to opt in to giving consent (please detail below)).

No: If the user is not asked for consent to use data for marketing separately.

No: If the user has not been required to provide a positive affirmative action, separate from accepting other T’s & C’s / Privacy Policies, to agree to sharing their data for the purposes of marketing.

Logic

DISABLED LOGIC - Disabled if DP10 does not contain Marketing, or if DT13 is answered as non-personal.

Scoring Impact

High risk applied if no + multiplier based on the nature of the data.

Is the user informed of how they can opt out of each of these activities?

ORC_D28

 Further Information

Guidance/Context

The list of activities can be found in question ORC_DP10. The developer should state how a user can opt out of each of these processing activities.

Response Type

Yes/No

Answer Criteria

Yes: If the app has an option to opt out/turn off data collection for external research or provides a contact email to get data removed from a study.

Yes: If the policy clearly explains to user how they can contact the developer to opt-out of all  sharing/processing activities.

No: If shared for any other reasons other than legal obligations and no option to opt out (email address in policy explicitly stating how to opt out or sliders within app).

No: If the policy only mentions how the user can opt out of one, but not all, activities.

Logic

DISABLED LOGIC - Disabled if D39a and D39b have not been answered yes.

Scoring Impact

Medium risk applied + multiplier based on the nature of the data. Risk can not be applied to both D28 and DP13.

If the user can not opt out of all processing activity, does the developer clearly explain which activities they cannot opt out of and why?

ORC_DP13

 Further Information

Guidance/Context

N/A

Response Type

Yes/No

Answer Criteria

Yes: If only shared for legal obligations - Policy must state who they will share the data with and for what legal purposes (e.g. protect rights, copyright).

Yes: If the developer has clearly set out justifiable reasons for not being able to deal with particular requests with regards to stopping certain processing/sharing activities.

No: If users are not informed of how they can either opt out of processing and sharing activities AND there is no justification from the developer as to why users cannot opt out of certain activities.

Logic

DISABLED LOGIC - Disabled if D39a and D39b have not been answered yes or D28 has been answered Yes.

Scoring Impact

Medium risk applied if no + multiplier based on the nature of the data. Risk can not be applied to both D28 and DP13.

Is the user informed that their data will not be shared with other parties, except for the purposes that have been set out in the privacy policy?

ORC_D16

 Further Information

Guidance/Context

Developers are required to share who data may be shared with for processing and other activities. Information for this question will typically be located in the privacy policy around information about what third parties data is shared with.

Response Type

Yes/No

Answer Criteria

Yes: If no data is shared without user consent.

Yes: If the policy states that using the app indicates agreement to the policy/given consent for data sharing specified.

No: If data is shared with third parties without user consent.

Logic

DISABLED LOGIC - Disabled if D39a has not been answered yes or D01 has been answered No or if DT13 has been answered non-personal.

Scoring Impact

High risk applied if no + multiplier based on the nature of the data.

 

Data Storage and Transit/Transfer

The key areas in this section are surrounding data storage and data transfer. The data privacy policy should inform the user of where their data is stored, how their data is protected in storage, and how it is protected in transit between the user’s device and the host storage. The NordDEC looks for specific and secure storage techniques, such as encryption or firewalls. During transit, it is preferable that data is protected using SSL encryption.

Question

Question Reference Source

Does the data privacy policy or equivalent provide detail about where the data collected by the app will be stored (i.e. on the app or in an external data warehouse, cloud server etc.)?

ORC_DST01

 Further Information

Guidance/Context

N/A

Response Type

Yes/No

Answer Criteria

Yes: If the policy states the data is stored in a cloud server e.g. physical location e.g. secure server.

Microsoft Azure is a cloud storage technology.

AWS - amazon web servers.

If policy states the physical address of the data controller.

“May not be stored in your location” isn’t enough.

“In the UK” isn’t enough, has to be an address.

Logic

DISABLED LOGIC - Disabled if D39a And D39b have not been answered yes.

Scoring Impact

Medium value applied if yes.

Where is the data stored? 

ORC_DST02

 Further Information

Guidance/Context

The purpose of this question is to state where data is stored e.g. in a secure server or an AWS server.

Response Type

Free Text

Answer Criteria

N/A

Logic

DISABLED LOGIC - Disabled if DST01 is no.

Scoring Impact

There is no scoring impact for this question.

Does the data privacy policy, or equivalent, state whether personal data is stored using recognised secure data storage technologies?

ORC_DST03

 Further Information

Guidance/Context

This question is looking to identify if appropriate technologies are being used for secure storage of user data. Technologies that are considered appropriate are detailed in the answer criteria below.

Response Type

Yes/No

Answer Criteria

Yes: If firewall, antivirus, or encryption when in storage/at rest is mentioned. Must state which technology is used, this does not have to be specific.
If AWS, Microsoft Azure or Google cloud server is mentioned.
“256-bit SSL encryption for data transfer and NSA-level 256-bit AES encryption of personal information” - would be “yes” to both encryption in storage and transit.
AES is yes to storage and encryption.
LUKS (Linux Unified Key Setup) is yes.

No: Doesn’t state which technology is used. Only mentions “cloud services” but doesn’t specify provider.

Logic

DISABLED LOGIC - Disabled if D39a And D39b have not been answered yes.

Scoring Impact

High value if yes. High risk if no.

Is all personally identifiable data encrypted in transit between the device and any external host storage?

ORC_D17

 Further Information

Guidance/Context

The purpose of this question is to ensure that data is transferred securely to ensure there are no breaches of users data.

Response Type

Yes/No

Answer Criteria

Yes: If the policy states encrypted during transit or mentions the encryption type.
Encryption types:
If SSL or TLS = encryption in transit.
HTTPS
Web apps - check the address bar, the padlock means HTTPS.
“256-bit SSL encryption for data transfer and NSA-level 256-bit AES encryption of personal information” - would be “yes” to both encryption in storage and transit.
AES (Yes to both storage and encryption).

No: Doesn’t state the data is encrypted during transit.
Only mentions payment details are encrypted in transit.

Logic

DISABLED LOGIC - Disabled if D39a has not been answered yes.

Scoring Impact

High risk applied if no.

Is the user informed that online video consultations use secure encryption methods?

ORC_DST04

 Further Information

Guidance/Context

Developers need to state that video consultations use secure encryption clearly if appropriate. This should be in addition to the previous question about encryption of other data transfer.

Response Type

Yes/No

Answer Criteria

Yes: If it is made explicitly clear to the end-user, that a secure encrypted connection is used for all video consultations. This may be in the policy or elsewhere on the website/app.

Logic

DISABLED LOGIC - Disabled if D39a has not been answered yes.

Scoring Impact

High risk applied if no .

 

Data Standards and Management

The NordDEC will award additional points if an app developer is compliant with any recognised International Data Management Standards such as ISO 27001. The privacy policy should inform users of a data retention period, and a method for data destruction. The NordDEC also identifies whether the developer has a policy in place to deal with any data security breaches.

Question

Question Reference Source

Does the policy state its compliance with recognised International Data Management Standards?

ORC_DM01

 Further Information

Guidance/Context

Developers that are compliant with these international data standards are rewarded for compliance with best practice standards of data management.

Response Type

Multiple Option

ISO 27001
Cyber Essentials+
Cyber Essentials (https://www.ncsc.gov.uk/cyberessentials/search)
PCI DSS
Other

Answer Criteria

Yes: If there is a compliance sticker on their website. ISO 27001. (if any other ISOs/BSIs etc. are mentioned, please confirm the appropriateness of the standard for data management.

Yes: Needs to be the COMPANY that is ISO compliant, not the server where data is stored, particularly when the company/developer is the data controller.

No: If there is no evidence of ISO, BSI etc. compliance.

EU-US privacy shield does not count for this question.

If the server (e.g. AWS) is ISO compliant but there is no explicit statement to say the company is.

Logic

DISABLED LOGIC - Disabled if D39a and D39b have not been answered yes.

Scoring Impact

High value applied if yes.

Does the policy contain details of the length of time data is retained?

ORC_D19

 Further Information

Guidance/Context

N/A

Response Type

Yes/No

Answer Criteria

Yes: If the developer mentions any time period of data retention, even if it’s an indefinite amount of time.

Yes: If the developer states “We only keep your personal information for as long as it’s necessary for our original legitimate purpose for collecting it and for as long as we have your permission to keep it.”

No: If the only mention of data retention is provided where the developer informs users of the rights under GDPR.

No: Developers are obliged to separately inform users of their own policies and procedures regarding data retention in the event that a user has not exercised any of their rights to their data.

No: If the policy mentions a timeframe after which data may be stored in aggregate.

No: If the policy states “we may retain data for…”

Logic

DISABLED LOGIC - Disabled if D39a has not been answered yes.

Scoring Impact

Low risk applied if no.

Is there a statement containing details of a method for data destruction?

ORC_D20

 Further Information

Guidance/Context

N/A

Response Type

Yes/No

Answer Criteria

Yes: If the policy mentions how the data is deleted, if users haven’t exercised any user rights.

Yes: If the policy details that user data is deleted after a certain time period.

Yes: If users can delete or reset all data within the app, AND it deletes it from the server. Not if it clears the app but stays on the server.

Yes: If users are the data controller and the method of deletion is users contacting the developer to remove their data.

Yes: If the developer has detailed the process for anonymizing personally identifiable information after a given timeframe of inactivity.

No: If the only mention for deletion of data is provided where the developer informs users of the rights under GDPR. Developers are obliged to separately inform users of their own policies and procedures regarding the deletion of data if a user has not exercised any of their rights to their data.

No: If the only mention is removal of data for under 13s/minors.

Logic

DISABLED LOGIC - Disabled if D39a has not been answered Yes.

Scoring Impact

Low risk score applied if D20 has been answered No, + multiplier based on the nature of the data collected.

Is there a statement that sets out a process for managing data confidentiality breaches?

ORC_D21

 Further Information

Guidance/Context

Developers have an obligation to notify the relevant supervisory authority when certain data breaches happen. Developers should therefore have a clear internal procedure in place to help aid the decision-making about whether or not a breach needs to be reported to the supervisory authority or even the affected individuals.

Response Type

Yes/No

Answer Criteria

Yes: If users are informed that they can complain to the Information Commissioner's Office (ICO), if they believe that their data privacy rights have been breached.

Yes: If users are informed they can lodge a complaint with their local data protection authority, if they believe that their data privacy rights have been breached.

Yes: If users are informed that they can complain to the Local Supervisory Authority of the country that the developer is based, if they believe that their data privacy rights have been breached.

Yes: If the user is told that they should inform the company, or the company will inform the user, (if you suspect a breach) and users have the right to file a complaint with the competent supervisory authority (GDPR Art. 77). Check T&Cs too.

Yes: If the developer has detailed in the privacy policy how they will approach any breaches to data security that they become aware of. For example informing users within a reasonable time frame and informing their relevant jurisdictional supervisory authority.

Yes: If the developer has detailed the process in which they will inform the local/jurisdictional regulatory authority of any confidentiality breaches.

No: If the policy doesn’t state what happens in the event of a breach.

No: If you can complain but only to the developer, not to the ICO in the event of a breach.

Logic

DISABLED LOGIC - Disabled if D39a has not been answered Yes.

Scoring Impact

Medium risk applied if D21 has been answered No + multiplier based on the nature of the data collected.

 

General Data Protection Regulation (GDPR)

This evaluation area focuses on the General Data Protection Regulation, which in May 2018 came into force. The NordDEC is concerned that all apps, particularly those developed in the EEA, are fully compliant with the GDPR. This means a clear and explicit statement of compliance, as well as confirming that the user is entitled to the 7 user rights.

The developer should also inform the user of how they can exercise these rights, and should commit to responding within a time frame of 2 months or less. Under the GDPR, the policy should outline the legal basis for collection of user data, and ensure that only minimal data is collected from the user.

All question relating to GDPR will only be asked for apps that collect and process personal and/or sensitive data.

Question

Question Reference Source

Is there a statement that confirms the App’s compliance to GDPR 2018?

ORC_D23

 Further Information

Guidance/Context

This statement may not always be situated within the policy. Check the terms and conditions and also other pages of the web site before answering this question no.

Response Type

Yes/No

Answer Criteria

Yes: If the developer states that they are fully compliant with GDPR or data protection act 2018. Or, are fully compliant to all 5 statements and states the GDPR/EU laws ((EU) 2016/679).

Yes: If the developer is registered with ICO - check registration number on ICO check website and provide a compliance statement.

No: If UK policies are not mentioned.

EU-US SHIELD is not GDPR compliant.

1998 data protection act is out of date.

Logic

DISABLED LOGIC - Disabled if D39a has not been answered Yes.

Scoring Impact

Low risk score applied if D23 has been answered No + with multiplier based on the nature of the data collected.

Is the user informed of the legal basis for which data is collected from them?

ORC_D60

 Further Information

Guidance/Context

To meet the requirement for this question the developer has to specify ‘the legal basis for data collection is…’

Response Type

Yes/No

Answer Criteria

Yes: If the policy states data is collected under a legal basis, e.g. consent, performance of contract, legal obligation, vital interests, public interest or legitimate interest.

Yes: If the policy states that by using the app you consent to the privacy policy, “If you consent to this app, you consent for us to collect data.”, or a statement similar to this.

Logic

DISABLED LOGIC - Disabled if D39a has not been answered Yes.

Scoring Impact

High risk score applied if D60 has been answered No + multiplier based on the nature of the data collected.

Is the user informed of the developer’s intent to ensure that data minimisation principles are met?

ORC_DPR03

 Further Information

Guidance/Context

Data minimisation means collecting the minimum amount of personal data that a developer needs to deliver an individual element of the service.

Response Type

Yes/No

Answer Criteria

Yes: If the policy mentions that only minimum data items necessary to provide their services are collected.

No: If there is no statement of a commitment to only collect necessary data from users.

No: If they have a statement to say they only collect the necessary health data - this statement doesn’t cover other types of data e.g. contact details.

Logic

DISABLED LOGIC - Disabled if D39a has not been answered Yes.

Scoring Impact

Low risk score applied if DPR03 has been answered No + multiplier based on the nature of the data collected.

Is there a statement that the policy will be updated duly should the purpose of data collection change? This may mean re-obtaining consent (if consent was the lawful basis).

ORC_D61

 Further Information

Guidance/Context

The developer has an obligation to inform users of any changes that are made to the processing of data. The level to which the developer must notify is determined by the legal basis for processing and the extent of the change being made.

Response Type

Yes/No

Answer Criteria

Yes: If the legal basis is consent and the developer states that if the purpose for processing data changes then consent will be re-obtained before continued use of the service. 

Yes: If consent is not one of the legal basis and the developer has stated in the privacy policy that they WILL inform users of changes to the policy. 

No: If the developer states that they MAY inform the users of changes to the privacy policy.

Logic

DISABLED LOGIC - Disabled if D39a has not been answered Yes.

Scoring Impact

Medium risk score applied if D61 has been answered No + multiplier based on the nature of the data collected.

Are users informed of their rights with regards to their data? Are users clearly informed of the individual privacy rights they are entitled to expect under GDPR?

ORC_DPR01

 Further Information

Guidance/Context

Questions relating to GDPR will only be asked for apps that collect and process personal and/or sensitive data.

Response Type

Yes/No

Answer Criteria

Yes: If the developer has made it clear that the user has certain rights with regards to their data and explains what those rights are.

Yes: If the developer has set out any of the user rights under GDPR.

Logic

DISABLED LOGIC - Disabled if D39a has not been answered Yes.

Scoring Impact

Exceptional risk score applied if DPR01 has been answered No + multiplier based on the nature of the data collected.

Has the developer made the existence of the data subject’s right to request that their personal data is deleted clear?

ORC_D93

 Further Information

Guidance/Context

This is a right users should expect under GDPR.

Response Type

Yes/No

Answer Criteria

Yes If the policy clearly states the user’s right to erasure, or method for how data is deleted, OR the user can clear all data from the app.

Yes: If the policy clearly states the user’s right to be forgotten.

Logic

DISABLED LOGIC - Disabled if D39a Or DPR01 have not been answered Yes.

Scoring Impact

Low value score applied if D93 has been answered Yes.

Has the developer made the existence of the data subject’s right to access their personal data clear?

ORC_D25

 Further Information

Guidance/Context

This is a right users should expect under GDPR.

Response Type

Yes/No

Answer Criteria

Yes: If the policy clearly states the user’s right to access their data, and a contact method is given.

Logic

DISABLED LOGIC - Disabled if D39a Or DPR01 have not been answered Yes.

Scoring Impact

Low value score applied if D25 has been answered Yes.

Has the developer made the existence of the data subject’s rights to rectify their personal data clear?

ORC_D56

 Further Information

Guidance/Context

This is a right users should expect under GDPR.

Response Type

Yes/No

Answer Criteria

Yes: The policy clearly states the user’s right to rectify, correct, amend or update their information.

Logic

DISABLED LOGIC - Disabled if D39a Or DPR01 have not been answered Yes.

Scoring Impact

Low value score applied if D56 has been answered Yes.

Has the developer made the existence of the data subject’s rights to restrict the use of their personal data clear?

ORC_D81

 Further Information

Guidance/Context

This is a right users should expect under GDPR.

Response Type

Yes/No

Answer Criteria

Yes: If the policy clearly states the user’s right to restrict use, or to stop using data.

Logic

DISABLED LOGIC - Disabled if D39a Or DPR01 have not been answered Yes.

Scoring Impact

Low value score applied if D81 has been answered Yes.

Has the developer made the existence of the data subject’s rights to object to the processing of their personal data clear?

ORC_D57

 Further Information

Guidance/Context

This is a right users should expect under GDPR.

Response Type

Yes/No

Answer Criteria

Yes: If the policy clearly states the user’s right to object.

Logic

DISABLED LOGIC - Disabled if D39a Or DPR01 have not been answered Yes.

Scoring Impact

Low value score applied if D57 has been answered Yes.

Has the developer made the existence of the data subject’s rights to portability of (receive) their personal data clear?

ORC_D59

 Further Information

Guidance/Context

This is a right users should expect under GDPR.

Response Type

Yes/No

Answer Criteria

Yes: If the policy clearly states the user’s right to portability, or right to transfer their data, or the right to receive their data in a machine-readable format.

Logic

DISABLED LOGIC - Disabled if D39a Or DPR01 have not been answered Yes.

Scoring Impact

Low value score applied if D59 has been answered Yes.

Has the developer made the existence of the data subject’s right to withdraw consent for the use of their personal data clear?

ORC_D58

 Further Information

Guidance/Context

This is a right users should expect under GDPR.

Response Type

Yes/No

Answer Criteria

Yes: If the policy clearly states the user’s right to withdraw consent.

Logic

DISABLED LOGIC - Disabled if D39a Or DPR01 have not been answered Yes.

Scoring Impact

Low value score applied if D58 has been answered Yes.

Has the developer made clear the existence of the user’s right to request that they are not subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her?

ORC_DPR02

 Further Information

Guidance/Context

Automated processing is what can occur when applying for things such as insurance, finance, mortgage etc. It gives an output which is based on details entered. The result would be a machine driven decision or figure of cost etc. Users have the right to request any such decision be reviewed by a human.

Response Type

Yes/No

Answer Criteria

Yes: There may be a simpler statement, such as “You have the right to request that we do not process your personal data for the purpose of automated decision making”.

Yes: If the developer has made clear that the user has this right under GDPR, even if they do not specifically process data in such a way.

No: If this user right has not been mentioned in the policy.

Logic

DISABLED LOGIC - Disabled if D39a Or DPR01 have not been answered Yes.

Scoring Impact

Low value score applied if DPR02 has been answered Yes.

Does the developer provide details which the user can contact them on to exercise their rights?

ORC_D82

 Further Information

Guidance/Context

N/A

Response Type

Yes/No

Answer Criteria

Yes: If a contact method is provided in the policy for the developer, in relation to exercising user rights.

No: If a contact method is only provided for one user right, rather than all rights mentioned.

Logic

DISABLED LOGIC - Disabled if D39a Or DPR01 have not been answered Yes.

Scoring Impact

Low value score applied if D81 has been answered Yes.

Is the user informed of the time frame in which the developer will respond to any requests to exercise their rights?

ORC_D83

 Further Information

Guidance/Context

N/A

Response Type

Yes/No

Answer Criteria

Yes: If a time frame is given, and it is within two months of receipt of the request.

No: If there is no separately provided timeframe and response commitment provided with regards to the user exercising their rights. I.e. if there are only contact details for enquiries about the policy as a whole, with an expected response time.

Logic

DISABLED LOGIC - Disabled if D39a Or DPR01 have not been answered Yes.

Scoring Impact

Low value score applied if D81 has been answered Yes.

 

Other Data Questions

This section also looks into children’s data use (if applicable), or if a user can report knowledge of a child accessing the app without parental consent. The transparency of the privacy policy should extend to inform the user that any links to third party websites or apps are not covered by the developer’s privacy policy, and users should make themselves aware of such third party policies. The privacy policy should contain contact details, should the user wish to make further enquiries regarding their data. The NordDEC also awards additional value points if the app provides the user with an additional, optional layer of security to protect their data.

 

Question

Question Reference Source

Are users clearly informed of the use of cookies when first landing on the developers site/app?

ORC_D99

 Further Information

Guidance/Context

When reviewing a native (including “hybrid”) app, being informed of the website using cookies, while using the browser, does not answer this question as yes. Reference to “site” in the question is regarding a review of a web app.

Response Type

Yes/No

Answer Criteria

Yes (web app): If when first landing on the website a banner/pop up appears asking users to accept cookies.

Yes (Native Apps): If when first accessing the app, or at the point at which the app attempts to use cookies, the user is clearly informed of the intended use of cookies.

Logic

DISABLED LOGIC - Disabled if the app is not collecting cookies, D01 is answered No or DT10 does not contain “Cookies/Web Beacons”

Scoring Impact

Low risk if answered No + Multiplier based on the nature of the data.

Are user's required to confirm their acceptance of the developer's use of cookies, when initially informed of the use?

ORC_D100

 Further Information

Guidance/Context

It is required that developers gain consent from visitors to the site in order to store or retrieve any information on a computer, smartphone or tablet using cookies.

Response Type

Yes/No

Answer Criteria

Yes (web app): If when first landing on the website a banner/pop up appears asking users to accept cookies.

Yes (Native Apps): If, when users are informed of the use of cookies, they are required to provide a clear confirmation of their acceptance of the use of cookies.

Logic

DISABLED LOGIC - Disabled if the app is not collecting cookies, D01 is answered No or DT10 does not contain “Cookies/Web Beacons”

Scoring Impact

Low risk if answered No + Multiplier based on the nature of the data.

Does the developer provide a full Cookie Policy, separate from the Terms of Service and/or Privacy Policy?

ORC_D84

 Further Information

Guidance/Context

Typically found on the bottom banner of a website or link. This should be made available when accepting cookies

Response Type

Yes/No

Answer Criteria

N/A.

Logic

DISABLED LOGIC - Disabled if the app is not collecting cookies, D01 is answered No or DT10 does not contain “Cookies/Web Beacons”.

Scoring Impact

Low risk if answered No + Multiplier based on the nature of the data.

Is the app ‘particularly likely’ to be used by children, even if they are not the primary market for the product?

ORC_D44

 Further Information

Guidance/Context

N/A

Response Type

Yes/No

Answer Criteria

Yes: The app has content which may be appealing to children, but doesn’t specify an age range, OR the app is intended to be used by children. E.g. Apps that have a reward system that allow users to build up coins that can be exchanged for rewards.

No: If policy states they won’t collect data from under 13s, GDPR still allows for the 13-16 age range to provide consent independently

No: The policy is quite clear that the app is aimed at people over 18 or they won’t take data from an under 18 AND the app does not present any particular content or features that would encourage a minor to attempt to access and use the app.

Logic

DISABLED LOGIC - Disabled if D01 is no, Or DS06 is answered yes, Or if DT13 is answered none personal.

Scoring Impact

None

Are users informed of how they can report, to the developer, any knowledge of a child accessing the app and providing personal data, without parental consent?

ORC_DO01

 Further Information

Guidance/Context

N/A

Response Type

Yes/No

Answer Criteria

Yes: If there is a statement that specifically details what a user should do to inform the developer.

Yes: If the developer specifically states that if they become aware of a minor/child providing personal data, then they will delete this data within a set period of time.

Yes: If the developer explains the app is offering online preventive or counselling services to children and therefore does not obtain parental consent as they are legally obliged to not do so.

No: If the policy does not provide any details on how the developer and/or user should respond when they become aware of a minor/child providing personal data.

Logic

DISABLED LOGIC - Disabled if D01 is no, Or DS06 is answered yes, Or if DT13 is answered none personal.

Scoring Impact

High, Medium or Low risk applied depending on if D44 is answered yes, if the app is designed for child, Pre-teen or teen with a multiplier applied based on the level of data collected (Non-personal, Personal, Sensitive)

Is the user made aware that by following links to third party websites, the developer’s policies no longer apply, and that the user should make themselves aware of the third party’s policies?

ORC_D91

 Further Information

Guidance/Context

Developers should make users aware that they should make themselves aware of third party policies as the developers privcay policy no longer applies. This may also be found in the Terms & Conditions.

Response Type

Yes/No

Answer Criteria

Yes: The policy mentions that the developer’s policy doesn’t extend to third parties and users are advised to make themselves aware of the privacy policies of any third party site/platform that they visit through the app.

Yes: Users are provided links to relevant third party privacy policies.

Logic

DISABLED LOGIC - Disabled if D39a And D39b have not been answered Yes.

Scoring Impact

Medium value score applied if D91 has been answered Yes.

Is the user informed of how they can make further enquiries about the company’s privacy policy?

ORC_D92

 Further Information

Guidance/Context

This question is looking to capture if the user has the ability to contact developers if they have questions around the privacy policy and their processes.

Response Type

Yes/No

Answer Criteria

Yes: The user is informed of/given a method of contact for any queries regarding the policy. Contact method must be in the policy, normally found toward the end of the policy.

No: If it says “contact us” but no contact method is given in the policy.

No: If there is no clear statement that directs the user to contact information for the purpose of enquiring about the company's privacy policy/practices.

Logic

DISABLED LOGIC - Disabled if D39a And D39b have not been answered Yes.

Scoring Impact

Medium value score applied if D92 has been answered Yes.

Does the app allow the user to set their preferences for sharing the app data with or from other apps (e.g. Facebook / Instagram/Fitbit etc)?

ORC_D06

 Further Information

Guidance/Context

This question looks to capture any sharing preferences includes sharing with third parties, not necessarily other apps.

Response Type

Yes/No

Answer Criteria

Yes: If the app allows user controls over data sharing with individual apps/platforms. I.e. gives the choice to turn on/off sharing with google fit, Instagram, Fitbit etc.

Yes: If it allows sign up through Facebook/Google+ or separately with an email address, as this gives the option to sign up with or without sharing with/from other apps/platforms.

No: If the app doesn’t ask permission to share to other apps, but does so automatically. Even if this is based on user agreement to privacy policies/T’s & C’s.

No: If ONLY sign up with Facebook/Google+ is allowed, i.e. you have no choice but to do so.

Logic

DISABLED LOGIC - Disabled if DS03 is answered No.

Scoring Impact

Medium value score applied if D06 has been answered Yes.

Is there functionality within the app to allow the user to set their preferences for sharing app data with others users (clinicians, carer, family, friends, buddies)?

ORC_D27

 Further Information

Guidance/Context

This question is looking to identify if users can choose if information is shared with other users. Added control over users own data is beneficial for maintaining privacy where appropriate.

Response Type

Yes/No

Answer Criteria

Yes: Can choose WHAT is being shared with WHO on the app.

Yes: If the data is only ever shared with other users through manual user intervention. E.g. users choose to post on a forum/news feed.

No: The app gives no control over who sees what. E.g. an open forum/send to all clinicians.

Logic

DISABLED LOGIC - Disabled if DS03is answered No.

Scoring Impact

Medium value score applied if DS03 has been answered Yes.

Is it strictly necessary for anyone to easily access the personal information that persists on the app? e.g. to access health info during an emergency.

ORC_DO02

 Further Information

Guidance/Context

This question is specific to the app on the device it has been downloaded onto. e.g Is there a need to access information stored on the app during an emergency.

Response Type

Yes/No

Answer Criteria

Yes: If the intended purpose of the app is to provide information to those providing emergency response, in the event that the individual concerned is unable to communicate with the responder.

No: If access is for any other reason, including if access is remote for clinicians to monitor patients.

Logic

DISABLED LOGIC - Disabled if DS03 is answered No.

Scoring Impact

Medium value score applied if DS03 has been answered Yes.

Are users provided options to introduce additional security measures to protect their data on the app? eg. set additional pass codes for access to the app, after accessing the device is unlocked.

ORC_DO03

 Further Information

Guidance/Context

Data contained within an app may be private to an individual. Adding security features to the app itself reassures users.

Response Type

Yes/No

Answer Criteria

Yes: App allows a pass code to be set, or use device security/unlock mechanisms a second time to access the app.

Yes: If there is the option to choose who can see information contained within the user profile/set privacy controls on you account.

No: If the user can set data sharing controls, such as choosing apps to share data with.

Logic

DISABLED LOGIC - Disabled if DO02 has been answered Yes.

Scoring Impact

High value score applied if DO03 has been answered Yes.

Does the app use a sign up/sign in verification/authentication model

ORC_DO04

 Further Information

Guidance/Context

This question is looking to identify if the users access/identify is verified in any way. This is important to ensure the person creating the account is who they say they are and has access to the related accounts e.g. Email address.

Response Type

Yes/No

Answer Criteria

Yes: If there are any forms of user authentication being used.

No: If the developer does not have any way, beyond signing in, by which they verify that the person creating/accessing an account, is the person that they claim to be/the owner of the account

Logic

DISABLED LOGIC - Disabled if DT14 has been answered no.

Scoring Impact

High value score applied if DO04 has been answered Yes.

What type of model is being used? (please describe)

ORC_DO05

 Further Information

Guidance/Context

N/A

Response Type

Multiple Option

Answer Criteria

None

One-step email authentication - if already signed up, check by resetting password, if email link is sent to reset it is this one.

Other one-step authentication - e.g. Biometric access, pin number

HCP Granted Access/Invite - A referral code needed to access the app which comes from the HCP.

Admin Granted Access/Invite - Healthcare provider granting access to each individual HCP/user.

SMS authentication - code sent to phone confirming it is you signing in

Two-step authentication - Use of a separate authenticator app or a code sent to phone/email whenever you sign into the app which needs to be confirmed. The app uses a second authentication step after the user has clicked an email verification link when signing up e.g. requests a mobile number and sends a verification code by text.

Multi-step authentication - any more than 2 steps

Qualification/HCP Registration Check - Being able to register as a clinician and having your credentials checked before being accepted as a HCP to provide information to patients.

Identification Check (Eg. drivers licence, passport) - scan/take photos of ID for sign up purposes or ID verification e.g. NHS app when signing up or for a clinician to verify the person they are speaking to is the patient they are supposed to be dealing with.

Logic

DISABLED LOGIC - Disabled if DT14 has been answered no.

Scoring Impact

None

 

Enhanced Data Evaluation

Question

Question Reference Source

Is the developer of the app the Data Controller, Data Processor or Product Manufacturer Only?

ORC_ERC_EDC_O01

 Further Information

Guidance/Context

This questions is needed to help guide what is expected of the developer in terms of completing a DPIA and what would be expected within the following questions.

Response Type

Multiple Option

Answer Criteria

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is the developer subject to EU Data Protection Laws?

ORC_ERC_EDC_O02

 Further Information

Guidance/Context

N/A

Response Type

Yes/No

Answer Criteria

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Provide EU data protection registration details:

ORC_ERC_EDC_O03

 Further Information

Guidance/Context

N/A

Response Type

Free Text

Answer Criteria

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

If this information is unavailable, please explain why?

ORC_ERC_EDC_O04

 Further Information

Guidance/Context

N/A

Response Type

Free Text

Answer Criteria

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is the developer of the App a public authority or body?

ORC_ERC_EDC_DPO01

 Further Information

Guidance/Context

N/A

Response Type

Yes/No

Answer Criteria

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Do any of the developer’s processing activities include automated systematic and extensive profiling with significant effects?

ORC_ERC_EDC_DPA01

 Further Information

Guidance/Context

This question can be answered once the following breakout questions have been determined. This question is to help determine the need for a DPIA.

Response Type

Yes/No

Answer Criteria

Has a systematic description of the intended data processing been provided?

Article 35(3) sets out three types of processing which always require a DPIA, one of which is Systematic and extensive profiling with significant effects.

“any systematic and extensive evaluation of personal aspects relating to natural persons which is based on automated processing, including profiling, and on which decisions are based that produce legal effects concerning the natural person or similarly significantly affect the natural person.” - See Article 35(3)

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Do any of the developer’s processing activities include large scale use of sensitive/special category data?

ORC_ERC_EDC_DPA02

 Further Information

Guidance/Context

This question is to help determine the need for a DPIA.

Response Type

Yes/No

Answer Criteria

Article 35(3) sets out three types of processing which always require a DPIA, one of which is Large scale use of sensitive data.

“processing on a large scale of special categories of data referred to in Article 9(1), or of personal data relating to criminal convictions and offences referred to in Article 10.” - See Article 35(3)

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Do any of the developer’s processing activities include systematic monitoring of a publicly accessible area on a large scale?

 ORC_ERC_EDC_DPA03

 Further Information

Guidance/Context

This question is to help determine the need for a DPIA.

Response Type

Yes/No

Answer Criteria

Article 35(3) sets out three types of processing which always require a DPIA, one of which is Public monitoring.

“a systematic monitoring of a publicly accessible area on a large scale.”

Systematic monitoring includes all forms of tracking and profiling on the internet, including for the purposes of behavioural advertising. - See Article 35(3)

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Do any of the developer’s processing activities include any of the following indicators of high risk processing?

 ORC_ERC_EDC_DPA04

 Further Information

Guidance/Context

If two or more of the answer options have been selected, then there is likely to be a need for the developer to have completed a DPIA, unless evidence can be provided that; it does not present risks to rights and freedoms of the data subjects; it is similar to another processing operation for which a DPIA has already been performed; A Member State law excludes the need of a DPIA; it is included in the optional list drawn up by the relevant supervisory authority pursuant to art. 35(5) GDPR.

In some cases, it is possible that just one of the options being selected will result in the need for a DPIA.

Response Type

Multiple Option

Answer Criteria

- Evaluation or scoring.
- Automated decision-making with legal or similar significant effect.
- Systematic monitoring.
- Sensitive data or data of a highly personal nature.
- Data processing/profiling on a large scale.
- Matching or combining datasets.
- Data concerning vulnerable data subjects.
- Innovative use or applying new technological or organisational solutions.
- Preventing data subjects from exercising a right or using a service or contract.
- Invisible processing.
- Risk of physical harm.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is the developer required to have a Data Protection Officer?

ORC_ERC_EDC_DPO06

 Further Information

Guidance/Context

Not all organisations are required to have a Data Protection Officer (DPO). This is determined by the type of organisation and core activities. The most common reason for organisations providing digital health technologies to have a DPO is due to the core activities involving processing health data (being a special category).

Response Type

Yes/No

Answer Criteria

The Information Commissioner has a self-assessment tool to determine whether you must appoint a DPO.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is the developer required to have written a Data Protection Impact Assessment (DPIA)?

 ORC_ERC_EDC_DPIA01

 Further Information

Guidance/Context

See ICO link for when a DPIA is needed. A DPIA is likely if at least 2 scenarios are present. Examples below (not exhaustive).

- When personal or sensitive data is processed
- If data processing may lead to high risk
- If automated profiling is involved
- High scale processing
- Processing data of vulnerable individuals.

Also refer to questions
ORC_ERC_EDC_DPA01
ORC_ERC_EDC_DPA02
ORC_ERC_EDC_DPA03
ORC_ERC_EDC_DPA04

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the developer submitted a DPIA or made a DPIA publicly available?

ORC_ERC_EDC_DPIA02

 Further Information

Guidance/Context

This question is needed as the following questions look to identify if the DPIA is accurate and robust.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the developer, at a minimum, published details on how to communicate with the DPO, for individuals to contact the DPO as needed?

ORC_ERC_EDC_DPO02

 Further Information

Guidance/Context

N/A

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Exceptional risk applied if answered no.

Is there evidence that the DPO has the necessary professional qualities, and in particular, experience and expert knowledge of data protection law?

ORC_ERC_EDC_DPO04

 Further Information

Guidance/Context

Necessary qualities would be that the DPOs must have a strong understanding of data protection law and regulatory requirements.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Exceptional risk applied if answered no.

Does the DPO hold a position within the organisation that may lead him or her to determine the purposes and the means of the processing of personal data, or require him or her to engage in further tasks and duties that may result in a conflict of interests with the primary tasks of a DPO?

ORC_ERC_EDC_DPO05

 Further Information

Guidance/Context

The DPO shouldn’t assume any other role in the organisation, other than that of Data Protection Officer. In this case, we may need to recommend that the DPO is a full time role, and to reconsider job titles.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Exceptional risk applied if answered no.

If a DPIA has been submitted, does the document provide details of a Data Protection Officer?

ORC_ERC_EDC_DPO03

 Further Information

Guidance/Context

Within a DPIA it would be expected that details of a DPO are shown to demonstrate they have the relevant credentials.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Exceptional risk applied if answered no.

Does the DPIA describe how the developer will collect data about individuals?

 ORC_ERC_EDC_DPA05

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the nature of the processing..

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details how data is collected about individuals, for example, input directly by the user, automatically collected, or from third party sources. This may be detailed in a data flow.

Logic

There is no disablement logic written for this question.

Scoring Impact
Low risk applied if answered no.

Does the DPIA describe how the developer will store the data collected?

 ORC_ERC_EDC_DPA06

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the nature of the processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details how the developer intends to store any data collected about an individual.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA describe what the data collected by the developer will be used for?

 ORC_ERC_EDC_DPA07

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the purpose of the processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details the intended use for the collected data.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA provide details of who will have access to the data collected?

 ORC_ERC_EDC_DPA08

 Further Information

Guidance/Context

This question is looking for clarity on who, internally, will have access to particular data, rather than naming organisations, where it is unclear who is a processor or controller. Addressing this question ensures that the DPIA is appropriate and robust surrounding the nature of processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details who will have access to the data collected.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA provide details of who the developer will share data with (any third parties)?

 ORC_ERC_EDC_DPA09

 Further Information

Guidance/Context

This question is looking for clarity on who, externally will have access to particular data. Addressing this question ensures that the DPIA is appropriate and robust surrounding the nature of processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details any third parties with whom individual’s data will be shared, as well as internally.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the information regarding data sharing include details of the developer’s use of any processors?

 ORC_ERC_EDC_DPA10

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the nature of the processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA detail the developer’s use of any processors.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA provide details of their own data retention periods?

 ORC_ERC_EDC_DPA11

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the nature of the processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details the developer’s own retention periods.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA provide details of the length of time each third party recipient of data will retain data for?

 ORC_ERC_EDC_DPA12

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the scope of the processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details the retention periods of all third party recipients of individual’s data.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA provide details of the security measures that have been put in place in order to protect the data being processed?

 ORC_ERC_EDC_DPA13

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the nature of the processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details how data is protected, eg. physical or electronical protection, who as access to data etc. Ideally this would be detail of procedures which have been put in place internally, such as staff training or limited access.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA provide details of the developer using any new technologies?

 ORC_ERC_EDC_DPA14

 Further Information

Guidance/Context

Innovative technology concerns new developments in technological knowledge in the world at large, rather than technology that is new to an individual, and its use can trigger the need to carry out a DPIA. This is because using such technology can involve novel forms of data collection and use, possibly with a high risk to individuals’ rights and freedoms. The personal and social consequences of deploying a new technology may be unknown, and a DPIA can help the controller to understand a control such risks.

Examples of processing using innovative technology include: artificial intelligence, machine learning and deep learning; connected and autonomous vehicles; intelligent transport systems; smart technologies (including wearables); market research involving neuro-measurement (e.g. emotional response analysis and brain activity);some ‘internet of things’ applications, depending on the specific circumstances of the processing.

Response Type

Multiple Option

Answer Criteria

Yes if the DPIA details whether or not any new technologies are in use.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA describe any novel types of processing that the developer is using?

 ORC_ERC_EDC_DPA15

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the nature of the processing.

Response Type

Multiple Option

Answer Criteria

Yes if the DPIA detail the developer’s use of any processors.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA describe which screening criteria they have identified/flagged as high risk?

 ORC_ERC_EDC_DPA16

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the nature of the processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA has identified any previous factors which may indicate likely high risk, and has flagged these in the DPIA.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA describe the nature of the personal data being processed?

 ORC_ERC_EDC_DPA17

 Further Information

Guidance/Context

This question is looking to capture whether the above questions have all been covered and the DPIA is appropriately and completely described the nature of the processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA provides a detailed description of the intended processing of data, the scope and purpose of the processing, reasons for purposing, and how the data is collected and processed.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA describe the volumes and variety of personal data that is being processed?

 ORC_ERC_EDC_DPA18

 Further Information

Guidance/Context

The UK GDPR does not contain a definition of large-scale processing, but to decide whether processing is on a large scale you should consider:
- the number of individuals concerned;
- the volume of data;
- the variety of data;
- the duration of the processing; and
- the geographical extent of the processing.

Examples of large-scale processing include:
- a hospital (but not an individual doctor) processing patient data;
- tracking individuals using a city’s public transport system;
- a fast food chain tracking real-time location of its customers;
- an insurance company or bank processing customer data;
- a search engine processing data for behavioural advertising; or
- a telephone or internet service provider processing user data.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details how many individuals are subject to data processing, and the variety of the data which is processed.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA describe the sensitive nature of any items of personal data being processed?

 ORC_ERC_EDC_DPA19

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the scope of the processing.

Response Type

Yes/No

Answer Criteria

Yes if any sensitive information is processed, and this is detailed within the DPIA.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA provide details of the extent and frequency of the processing?

 ORC_ERC_EDC_DPA20

 Further Information

Guidance/Context

Frequency cannot depend on the user keeping information up to date. It needs to be explicitly clear that processing will only occur when the user updates the info/adds an entry. The developer should therefore indicate how frequently each processing activity will be carried out.

The extent may be covered by the responses for the nature of the data collected and the processing purposes.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details the geographical extent of processing, and the frequency of processing. This comes under the scope of processing.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA provide details on the duration of any processing activities covered by the DPIA?

 ORC_ERC_EDC_DPA21

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the scope of the processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details the intended duration of any processing activities, or details any commitment to data minimisation principles, such as keeping data no longer than necessary, and removing data correctly. This covers the duration that processing occurs, and is different to the retention period. This comes under the scope of processing.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA detail the number of data subjects involved in the processing activities described?

 ORC_ERC_EDC_DPA22

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the scope of the processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details a definitive number, or indicates the number of individuals involved in data processing.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA provide details of the geographical area covered by the processing activities described?

 ORC_ERC_EDC_DPA23

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the scope of the processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA provides detail of the geographical area covered by the processing activities. This comes under the scope of processing. Spatial data is any data with a direct or indirect reference to a specific location or geographical area. Spatial data is often referred to as geospatial data or geographic information.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA identify all sources of the data being collected?

 ORC_ERC_EDC_DPA24

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the context of the processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details all sources where a developer may collect an individual’s data from, eg. directly from the user, or from third parties. This is to determine the context of processing.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA describe the nature of the developer’s relationship with the individuals whose data is being processed?

 ORC_ERC_EDC_DPA25

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the context of the processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details the relationship between the developer and the user of the app or product. Yes if the DPIA details that users are informed of data processing, for what purpose, and who with. This is to determine the context of processing.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA describe how far individuals have control over their data?

 ORC_ERC_EDC_DPA26

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the context of the processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details how far individuals have control over their data, specifically if it details how they can exercise control over their user rights.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA describe how far individuals are likely to expect the processing of their data to occur?

 ORC_ERC_EDC_DPA27

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the context of the processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details how far individuals can expect the processing of their data to occur. This is to determine the context of processing.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA provide details on whether the data subjects concerned include children or other vulnerable individuals?

 ORC_ERC_EDC_DPA28

 Further Information

Guidance/Context

Individuals can be vulnerable where circumstances may restrict their ability to freely consent or object to the processing of their personal data, or to understand its implications.

Most obviously, children are regarded as vulnerable to the processing of their personal data since they may be less able to understand how their data is being used, anticipate how this might affect them, and protect themselves against any unwanted consequences. This can also be true of other vulnerable sections of the population such as elderly people, or those with certain disabilities.

Even if the individuals concerned are not part of a group that would automatically be consider vulnerable, an imbalance of power in their relationship with the data controller can cause vulnerability for data protection purposes if individuals believe that they will be disadvantaged if the processing doesn’t go ahead.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA outlines whether the data subjects are likely to be children or other vulnerable individuals. The use of the personal data of children or other vulnerable individuals for marketing purposes, profiling or other automated decision-making, or if you intend to offer online services directly to children - this is likely to result in high risk.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA identify any previous experience that the developer has in dealing with the intended type of processing?

 ORC_ERC_EDC_DPA29

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the context of the processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details any previous experience the developer has, of this type of processing. Only relevant if there are new, innovative or unique means of processing.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA identify any relevant advances in the technology and/or security being used in the processing?

 ORC_ERC_EDC_DPA30

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the context of the processing.

Response Type

Multiple Options

Answer Criteria

Yes if the DPIA details any relevant advances in the technology and/or security used in the processing.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA identify any current issues of public concern with regards to the intended processing?

 ORC_ERC_EDC_DPA31

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the context of the processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details any current issues of public concern regarding the processing.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA detail compliances with any UK GDPR codes of conduct or UK certification schemes?

 ORC_ERC_EDC_DPA32

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the context of the processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details compliance with UK GDPR codes of conduct, or any other UK certification schemes for example ISO, CE etc.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA detail whether the developer has considered and complied with relevant codes of practice?

 ORC_ERC_EDC_DPA33

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the context of the processing.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA provide details of the developer’s legitimate interests, with regards to the purpose of processing data?

 ORC_ERC_EDC_DPA34

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the purpose of the processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details the developer’s legitimate interests.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA detail the legal basis/bases upon which the developer relies for processing the data collected?

 ORC_ERC_EDC_DPA35

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the purpose of the processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details any of the 6 accepted lawful/legal bases upon which the developer relies:

The lawful bases for processing are set out in Article 6 of the UK GDPR. At least one of these must apply whenever you process personal data:

(a) Consent: the individual has given clear consent for you to process their personal data for a specific purpose.

(b) Contract: the processing is necessary for a contract you have with the individual, or because they have asked you to take specific steps before entering into a contract.

(c) Legal obligation: the processing is necessary for you to comply with the law (not including contractual obligations).

(d) Vital interests: the processing is necessary to protect someone’s life.

(e) Public task: the processing is necessary for you to perform a task in the public interest or for your official functions, and the task or function has a clear basis in law.

(f) Legitimate interests: the processing is necessary for your legitimate interests or the legitimate interests of a third party, unless there is a good reason to protect the individual’s personal data which overrides those legitimate interests. (This cannot apply if you are a public authority processing data to perform your official tasks.)

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA provide details of the intended outcomes of the data processing for the individuals concerned?

 ORC_ERC_EDC_DPA36

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the purpose of the processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details the intended outcome of the data processing.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA provide details of the anticipated benefits of the processing for the developer or for society as a whole?

 ORC_ERC_EDC_DPA37

 Further Information

Guidance/Context

A key step of writing a DPIA is to describe the processing that will be undertaken. This must include the nature, scope, context and purposes of processing. Addressing this questions ensures that the DPIA is appropriate and robust surrounding the purpose of the processing.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details the expected benefits of processing, for them or for society as a whole.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Has the developer sought and documented the views of the individuals whose data will be processed, or their representatives?

 ORC_ERC_EDC_DPA38

 Further Information

Guidance/Context

Developers should seek and document the views of individuals (or their representatives) unless there is a good reason not to. See Step 3: Do we need to consult individuals?

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details whether the view of individuals (or their representatives) have been sought and documented - if this has not happened, the DPIA should then detail why not.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA provide a justifiable reason for not carrying out consultation with individuals or their representatives?

 ORC_ERC_EDC_DPA39

 Further Information

Guidance/Context

In most cases it should be possible to consult individuals in some form. However, if developers decide this is not appropriate, you should record this decision as part of your DPIA, with a clear explanation. For example it may compromise security. See Step 3: Do we need to consult individuals?

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA provide details of consultation with any third party data processors who will be involved in the processing of user data?

 ORC_ERC_EDC_DPA40

 Further Information

Guidance/Context

If developers use a data processor, there may be need to ask them for information and assistance. See Step 3: Do we need to consult anyone else?

Response Type

Multiple Options

Answer Criteria

Yes if the DPIA details any consultation with any third party processors.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA provide details of consultation with all relevant internal stakeholders?

 ORC_ERC_EDC_DPA41

 Further Information

Guidance/Context

Developers should consult all relevant internal stakeholders, in particular anyone with responsibility for information security.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA provides details of any consultation with relevant internal stakeholders. In particular, we would like to see evidence of consulting anyone with a responsibility toward Information Security / Information Governance.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA provide details of any advice being sought from other independent experts?

 ORC_ERC_EDC_DPA42

 Further Information

Guidance/Context

It is recommend that seeking legal advice or advice from other independent experts such as IT experts, sociologists or ethicists where appropriate. However, there are no specific requirements to do so

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details any advice from other independent experts.
The ICO recommends developers consider seeking legal advice or advice from other independent experts such as IT experts, sociologists or ethicists where appropriate. However, there are no specific requirements to do so.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA include details on how the developers plans will help achieve their purpose?

 ORC_ERC_EDC_DPA43

 Further Information

Guidance/Context

This question is needed to assess necessity and proportionality. This is a key step in ensuring the appropriateness and robustness of the DPIA.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details how the plans will help to achieve their purpose.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA provide evidence that the developer has considered whether there are any other reasonable ways to achieve the same result?

 ORC_ERC_EDC_DPA44

 Further Information

Guidance/Context

This question is needed to assess necessity and proportionality. This is a key step in ensuring the appropriateness and robustness of the DPIA.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details that the developer has considered alternative ways to achieve the same result. .

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA include details on how the developer will prevent function creep?

 ORC_ERC_EDC_DPA45

 Further Information

Guidance/Context

Function creep is defined as data being used for a purpose that is different to the original specified purpose. A DPIA may consider if the data use is compatible with the collection purpose, and if data is used for business processes or purposes other than the initial intended use.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA include details on how the developer will ensure data quality?

 ORC_ERC_EDC_DPA46

 Further Information

Guidance/Context

This question is needed to assess necessity and proportionality and how developers are ensuring data protection compliance. This is a key step in ensuring the appropriateness and robustness of the DPIA.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details for example what quality checks are in place to ensure data is accurate and up to date.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA include details on how the developer will ensure data minimisation?

 ORC_ERC_EDC_DPA47

 Further Information

Guidance/Context

This question is needed to assess necessity and proportionality and how developers are ensuring data protection compliance. This is a key step in ensuring the appropriateness and robustness of the DPIA. See Data minimisation.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details how data minimisation is ensured. This means ensuring that data processed is adequate, relevant, and limited to what is necessary.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA include details on how the developer intends to provide privacy information to individuals?

 ORC_ERC_EDC_DPA48

 Further Information

Guidance/Context

This question is needed to assess necessity and proportionality and how developers are ensuring data protection compliance. This is a key step in ensuring the appropriateness and robustness of the DPIA. See What privacy information should we provide.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details for example how the privacy policy is provided to individuals. This is to assess necessity and proportionality.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA include details on how the developer intends to implement and support individuals’ rights?

 ORC_ERC_EDC_DPA49

 Further Information

Guidance/Context

This question is needed to assess necessity and proportionality and how developers are ensuring data protection compliance. This is a key step in ensuring the appropriateness and robustness of the DPIA.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details how the developer intends to implement and support the user’s rights under GDPR.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA include details on the measures that will be taken to ensure any processors comply with their obligations?

 ORC_ERC_EDC_DPA50

 Further Information

Guidance/Context

This question is needed to assess necessity and proportionality and how developers are ensuring data protection compliance. This is a key step in ensuring the appropriateness and robustness of the DPIA.

Response Type

Multiple Option

Answer Criteria

Yes if the DPIA details how they will ensure any processors comply with their obligations through contracts in place.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA include details of the safeguarding measures that have been put in place for international transfers of data?

 ORC_ERC_EDC_DPA51

 Further Information

Guidance/Context

This question is needed to assess necessity and proportionality and how developers are ensuring data protection compliance. This is a key step in ensuring the appropriateness and robustness of the DPIA.

Response Type

Multiple Option

Answer Criteria

N/A if data is not transferred internationally.
Yes if data is transferred internationally, and the DPIA details any safeguarding measures in place.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Has the developer detailed, in the DPIA, all the risks that they have identified relating to each of their processing activities and those relating to third party processors?

 ORC_ERC_EDC_DPA52

 Further Information

Guidance/Context

This is a key step in ensuring the appropriateness and robustness of the DPIA. The developer should consider the potential impact on individuals and any harm or damage the processing may cause.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details all risks which have been identified or considered, relating to each processing activity.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Are there any additional risks, that you have been able to identify, which have not been included in the DPIA? (please detail)

 ORC_ERC_EDC_DPA53

 Further Information

Guidance/Context

Consider the potential impact on individuals and any harm or damage the processing may cause – whether physical, emotional or material. In particular, look at whether the processing could contribute to:
- inability to exercise rights (including but not limited to privacy rights);
- inability to access services or opportunities;
- loss of control over the use of personal data;
- discrimination;
- identity theft or fraud;
- financial loss;
- reputational damage;
- physical harm;
- loss of confidentiality;
- re-identification of pseudonymised data; or
- any other significant economic or social disadvantage.

Response Type

Multiple Choice

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA include an assessment of potential security risks?

 ORC_ERC_EDC_DPA54

 Further Information

Guidance/Context

To ensure the appropriateness and robust of the DPIA the developer should include sources of risk and the potential impact of each type of breach. e.g. illegitimate access

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details any security risks which have been considered, such as data breach, illegitimate access, modification or loss of personal data.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the assessment of security risks include the sources of the risks and the potential impacts of the each type of breach?

 ORC_ERC_EDC_DPA55

 Further Information

Guidance/Context

To

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details the sources of the risks considered, and the potential impact of each risk.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Has an objective approach to assessing the risks been taken through the use of a structured risk matrix that takes into account both the likelihood of harm and the severity of the impact?

 ORC_ERC_EDC_DPA56

 Further Information

Guidance/Context

N/A

Response Type

Yes/No

Answer Criteria

Yes, if there is evidence of a risk matrix within the DPIA, or it sets out both likelihood of harm and severity of impact, in a way that an objective assessment can be taken.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Has the developer used a different, but acceptable structured approach to objectively assessing the risks associated with their processing activities?

 ORC_ERC_EDC_DPA57

 Further Information

Guidance/Context

Examples of a structured matrix can be found on the ICO website. However appropriate alternatives may be used.

Response Type

Yes/No

Answer Criteria

Yes, if the DPIA gives any other way of objectively assessing risk, other than a risk matrix.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Has the developer also considered their own corporate risk, such as the impacts of regulatory action, reputational damage of loss of public trust?

 ORC_ERC_EDC_DPA58

 Further Information

Guidance/Context

This questions is needed to ensure the appropriateness and robustness of the DPIA.

Response Type

Yes/No

Answer Criteria

Yes, if there is any detail of considered risk in terms of regulatory action, reputational damage or loss of public trust.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Has the developer detailed the source of each risk that they have identified?

 ORC_ERC_EDC_DPA59

 Further Information

Guidance/Context

To ensure the appropriateness and robustness of the assessment of risk within the DPIA the developer needs to detail the source of each risk.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details the source of each identified risk.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Has the developer identified the measures that they will put in place in order to mitigate each of the identified risks?

 ORC_ERC_EDC_DPA60

 Further Information

Guidance/Context

To ensure the appropriateness and robustness of the assessment of risk within the DPIA the developer needs to detail the source of each risk. Once these risk have been identified the risks should look to be reduced where possible.

Response Type

Yes/No

Answer Criteria

Yes if there are any controls or mitigations in place, to mitigate against the listed risks.

For example:
- deciding not to collect certain types of data;
- reducing the scope of the processing;
- reducing retention periods;
- taking additional technological security measures;
- training staff to ensure risks are anticipated and managed;
- anonymising or pseudonymising data where possible;
- writing internal Guidance/Context or processes to avoid risks;
- using a different technology;
- putting clear data-sharing agreements into place;
- making changes to privacy notices;
- offering individuals the chance to opt out where appropriate; or
- implementing new systems to help individuals to exercise their rights.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Where mitigations have not been put in place, has the developer provided justified reasons for not doing so?

 ORC_ERC_EDC_DPA61

 Further Information

Guidance/Context

Where is it not appropriate to reduce the risks, it is necessary to provide justification. For example developers may take into account the cost and benefits of each measure.

Response Type

Yes/No

Answer Criteria

Yes if there are no mitigations in place, and the developer has provided a justification as to why.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Are there any additional mitigations, that you have been able to identify as achievable, that the developer does not appear to have considered? (please detail).

 ORC_ERC_EDC_DPA62

 Further Information

Guidance/Context

During the assessment process the assessor should look for any alternative mitigations that may be put in place by the developer that do not appear to have been considered in the documentation.

Response Type

Multiple Option

Answer Criteria

Yes, if it is possible to identify any additional mitigations which have not already been stated by the developer.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA contain details of the additional measures that the developer planned on taking?

 ORC_ERC_EDC_DPA63

 Further Information

Guidance/Context

To ensure the appropriateness and robustness of the assessment of risk within the DPIA the developer needs to detail the additional measures.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA contain a record on whether each of the risks have been removed, reduced, or accepted?

 ORC_ERC_EDC_DPA64

 Further Information

Guidance/Context

To ensure the appropriateness and robustness of the assessment of risk within the DPIA all risks need to have been mitigated against or accepted as appropriate. It may not be possible to eliminate every risk, and it could be decided that some risks, and even a high risk, are acceptable given the benefits of the processing and the difficulties of mitigation a developer may face.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details whether each of the risks have been removed, reduced or accepted.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA provide details of the overall residual risk, after taking additional measures?

 ORC_ERC_EDC_DPA65

 Further Information

Guidance/Context

To ensure the appropriateness and robustness of the assessment of risk within the DPIA all residual risks need to be stated.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details any residual risk, after additional measures and mitigations.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Have any of the residual risks been assessed as being “high risk”?

 ORC_ERC_EDC_DPA66

 Further Information

Guidance/Context

It is important to highlight any residual “high risks” as these requite consultation with the ICO before processing can continue.

Response Type

Yes/No

Answer Criteria

Yes if any of the residual risks are identified as ‘high risk.’

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA provide details that the need to consult the ICO has been considered?

 ORC_ERC_EDC_DPA67

 Further Information

Guidance/Context

To ensure the appropriateness and robustness of the assessment of risk within the DPIA all residual high risks must have been though consultation with the ICO before processing can continue.

Response Type

Yes/No

Answer Criteria

Yes if the DPIA details whether the need to consult the ICO has been considered.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the DPIA provide details or evidence of the advice of the DPO being sought, as part of the sign-off process?

 ORC_ERC_EDC_DPA68

 Further Information

Guidance/Context

To ensure the appropriateness and robustness of the DPIA, during sign-off advice from the DPO on whether the processing is compliant and can go ahead.

Response Type

Yes/No

Answer Criteria

Yes if the advice of the ICO has been sought, and the DPIA details evidence of this advice.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

If the developer has decided not to follow the advice provided by the DPO, have they recorded their reasons?

 ORC_ERC_EDC_DPA69

 Further Information

Guidance/Context

As part of the sign-off process for the DPIA, the developer should seek and document DPO advice on whether the processing is compliant and can go ahead. If they decide not to follow their advice, they need to record their reasons.

They should also record any reasons for going against the views of individuals or other consultees.

Response Type

Multiple Option

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Does the app process personal/sensitive Social Care data?

ORC_ERC_EDC_NHS01

 Further Information

Guidance/Context

This questions is needed to gather information on whether the app is processing social care data. Social care can be defined as the provision of social work, personal care, protection or social support services to children or adults in need or at risk, or adults with needs arising from illness, disability, old age or poverty. This helps guide the assessment in whether the developer is complying with all relevant standards. For instance Data Security & Protection Toolkit.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question

Does the developer collect usage or bug report data?

ORC_ERC_EDC_UBRO01

 Further Information

Guidance/Context

It can be beneficial to developers when they collect this data as it allows further insight into their product. If a developer does collect this information there are certain criteria that need to be met when they process it.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question

Is this collected through informed consent?

ORC_ERC_EDC_UBRO02

 Further Information

Guidance/Context

Informed consent is a process whereby the individual is fully informed of the nature and specific purpose of the project, the data that is being collected, and how that data will be used

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Low risk applied if answered no.

Is this data fully anonymised?

ORC_ERC_EDC_UBRO03

 Further Information

Guidance/Context

Anonymisation is the process of removing any personal identifiers, that may lead to an individual being identified. Usage data can still result in users being identified if not anonymised properly. Due to the application of usage data they may be stored in a less secure environment so sensitive data could be compromised if not anonymised before use.

Response Type

Yes/No

Answer Criteria

N/A.

Logic

There is no disablement logic written for this question.

Scoring Impact

High risk applied if answered no.

Does the organisation collect data, through the app, using cookies, web beacons or other similar technologies?

ORC_ERC_EDC_CK01

 Further Information

Guidance/Context

The use of Cookies, web beacons or other similar technologies require the developer to comply with certain regulations. This question helps identify what needs to be asked of the developer.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is there a cookie policy?

ORC_ERC_EDC_CK02

 Further Information

Guidance/Context

GDPR and most other privacy laws require that developers disclose what information is collected and what is done with this information. If cookies are used to collect data then the developer must have a cookie policy to explain what they are for. This can be added within the privacy policy as long as it is in a clear separate section.

Response Type

Yes/No

Answer Criteria

N/A.

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_CK01 is answered no.

Scoring Impact

High risk applied if answered no.

Is there a cookie policy provided separate from the terms and condition and privacy policy?

ORC_ERC_EDC_CK03

 Further Information

Guidance/Context

If a developer does not provide a separate cookie policy page they can describe the cookies and there purpose within the privacy policy. As long as the cookies section is clearly defined and sperate from the other sections of the privacy policy.

Response Type

Yes/No

Answer Criteria

N/A

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_CK01 is answered no.

Scoring Impact

There is no scoring logic for this question.

Are users made aware of the use of strictly necessary cookies?

ORC_ERC_EDC_CK04

 Further Information

Guidance/Context

App/Site owners must obtain consent from users before using any cookies, with the exception of strictly necessary cookies.

Response Type

Yes/No

Answer Criteria

N/A.

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_CK01 is answered no.

Scoring Impact

Low risk applied if answered no.

Is user consent obtained for the use of non-strictly necessary cookies?

ORC_ERC_EDC_CK05

 Further Information

Guidance/Context

App/Site owners must obtain consent from users before using any cookies, with the exception of strictly necessary cookies.

Response Type

Yes/No

Answer Criteria

N/A

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_CK01 is answered no.

Scoring Impact

Low risk applied if answered no.

Does the organisation keep a log of user consent? (eg. evidence of when consent was obtained and the information provided at the time of consent)

ORC_ERC_EDC_CK06

 Further Information

Guidance/Context

App/Site Owners must document and store when consent was obtained from users and should in addition keep a record of the information provided to a user when their consent was obtained.

Response Type

Yes/No

Answer Criteria

N/A

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_CK01 is answered no.

Scoring Impact

Low risk applied if answered no.

Are users informed of how they can easily opt out of the use of cookies?

ORC_ERC_EDC_CK07

 Further Information

Guidance/Context

After accepting the use of cookies it should be as easy for users to withdraw their consent for the use of cookies as it was for them to give their consent in the first place.

Response Type

Yes/No

Answer Criteria

N/A

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_CK01 is answered no.

Scoring Impact

Low risk applied if answered no.

Is the product aimed at children or likely to be used by children?

ORC_ERC_EDC_COP01

 Further Information

Guidance/Context

This question needs to be considered as children need particular protection as they may be less aware of the risks involved with their data being collected and processed.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Where consent was the legal basis for processing data was consent, at the time the an individual was a child, then requests for the erasure of data are complied with, whenever possible?

ORC_ERC_EDC_COP04

 Further Information

Guidance/Context

This question is asking that if a user is a child at the time the data was collected about them, does the developer ensure that in adulthood, they are able to exercise their rights with regards to their data. Eg. They don’t refuse rights because the data was not collected based on consent from the child as at the time of collection, but instead based upon parent/guardian consent.

Response Type

Yes/No

Answer Criteria

N/A

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_COP01 is answered no.

Scoring Impact

Low risk applied if answered no.

Have children been consulted when designing this processing practice?

ORC_ERC_EDC_COP05

 Further Information

Guidance/Context

N/A

Response Type

Yes/No

Answer Criteria

N/A

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_COP01 is answered no.

Scoring Impact

Low risk applied if answered no.

Has the privacy policy been written in plain, age appropriate language?

ORC_ERC_EDC_COP06

 Further Information

Guidance/Context

Privacy policies need to be written, or made available, in appropriate language so that children are able to understand what will happen to their data and what rights they have.

Response Type

Yes/No

Answer Criteria

N/A

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_COP01 is answered no.

Scoring Impact

Low risk applied if answered no.

Is consent sought from a responsible parent/guardian?

ORC_ERC_EDC_COP07

 Further Information

Guidance/Context

If consent is the legal bases for processing only certain children are able to provide their consent. Consideration into the law is needed for the requisite region. For example in the UK only children aged 13 or over are able to provide their own consent. For children under this age you need to get consent from whoever holds parental responsibility for the child - unless the online service you offer is a preventive or counselling service.

Response Type

Yes/No

Answer Criteria

N/A

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_COP01 is answered no.

Scoring Impact

There is no scoring impact associated with this question.

Does the developer ensure they do not seek parental/guardian consent when providing online preventive or counselling services to children?

ORC_ERC_EDC_COP08

 Further Information

Guidance/Context

If the app in question is providing online preventive or counselling services to children then the GDPR says that parental consent should not be required. In these scenarios it may be in the best interest of the child for safeguarding to accept their own consent, or that another basis for processing may be more appropriate.

Response Type

Yes/No

Answer Criteria

N/A

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_COP01 is answered no.

Scoring Impact

Medium risk applied if this question and ORC_ERC_EDC_COP07 are answered no.

Are there two separate versions of privacy policies, one aimed at the child and the other at the responsible parent/guardian?

ORC_ERC_EDC_COP09

 Further Information

Guidance/Context

N/A

Response Type

Yes/No

Answer Criteria

N/A

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_COP01 is answered no.

Scoring Impact

Low risk applied if this question and ORC_ERC_EDC_COP06 are answered no.

When marketing the product outside of their country of residence, has the developer taken into consideration other jurisdictional laws regarding children’s privacy (eg. age restrictions)?

ORC_ERC_EDC_COP10

 Further Information

Guidance/Context

It is important the developer takes into account any jurisdictional laws when marketing outside of their country of residence as different countries may have different laws regarding children and their data privacy.

Response Type

Yes/No

Answer Criteria

N/A

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_COP01 is answered no.

Scoring Impact

Medium risk applied if answered no.

Has the DPIA been completed with specific details on the assessed risks to children and the mitigations in place?

ORC_ERC_EDC_COP03

 Further Information

Guidance/Context

During the completion of the DPIA, consideration into the risks related to children needs to be specifically considered as they may differ to the risks related to adults.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

High risk applied if answered no.

Has a process been designed and put in place that allows children to easily access, understand and exercise their own data protection rights?

ORC_ERC_EDC_COP02

 Further Information

Guidance/Context

Children have the same rights as adults over their personal data. These include the rights to access their personal data; request rectification; object to processing and have their personal data erased.

Response Type

Yes/No

Answer Criteria

N/A

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_COP01 is answered no.

Scoring Impact

Low risk applied if answered no.

Is data shared with and processed by any third parties?

ORC_ERC_EDC_PC01

 Further Information

Guidance/Context

Data processing agreements come with certain obligations that fall upon the processors. It is important to know if a developer is sharing data with a third party and if so what process are in place to ensure this is done correctly.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

What services do the third parties provide?

ORC_ERC_EDC_PC02

 Further Information

Guidance/Context

This question seeks to identify the purpose of the third party and what service they provide. e.g. Payment processing.

Response Type

Multiple Choice

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is any of the shared data personally identifiable or sensitive?

ORC_ERC_EDC_PC03

 Further Information

Guidance/Context

Depending on the type of data that is shared will determine the expectations of what needs to be in place at the third party processors.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Are there written binding contract agreements between the organisation (controller) and each third party (processor)?

ORC_ERC_EDC_PC04

 Further Information

Guidance/Context

Data processing agreements come with certain obligations that will fall upon the processors. The contracts between the controller and the processor should outline what the controller has stated will happen with the DPIA. Contracts are put in place by the controller to ensure that data is handled appropriately.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Exceptional risk applied if answered no.

Has the developer provided a copy/draft/template of these written agreements?

ORC_ERC_EDC_PC05

 Further Information

Guidance/Context

This questions allows assessors to understand what steps the controller has put in place by viewing an example agreement. These can then be outlined and checked in the following questions.

Response Type

Yes/No

Answer Criteria

N/A

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_PC01 or ORC_ERC_EDC_PC04 are answered no.

Scoring Impact

Exceptional risk applied if answered no.

Does the contract clearly define the subject matter and duration of the processing?

ORC_ERC_EDC_PC06

 Further Information

Guidance/Context

It is the responsibility of the data controller to ensure that purpose and duration of the processing is made clear to the processor.

Response Type

Yes/No

Answer Criteria

N/A

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_PC05 is answered no.

Scoring Impact

Low risk applied if answered no.

Does the contract clearly define the purpose of the processing?

ORC_ERC_EDC_PC07

 Further Information

Guidance/Context

It is the responsibility of the data controller to ensure that the purpose and scope of the processing is made clear to the processor. This should align with what the data controller has outlined in the DPIA.

Response Type

Yes/No

Answer Criteria

N/A

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_PC01 or ORC_ERC_EDC_PC04 or ORC_ERC_EDC_PC05 are answered no.

Scoring Impact

Low risk applied if answered no.

Does the contract clearly set out the categories of data subject and the types of data that will be processed?

ORC_ERC_EDC_PC14

 Further Information

Guidance/Context

It is the responsibility of the data controller to ensure that the purpose and scope of the processing is made clear to the processor. This should align with what the data controller has outlined in the DPIA.

Response Type

Yes/No

Answer Criteria

N/A

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_PC01 or ORC_ERC_EDC_PC04 or ORC_ERC_EDC_PC05 are answered no.

Scoring Impact

Low risk applied if answered no.

Does the contract clearly describe the obligations and rights of the controller?

ORC_ERC_EDC_PC08

 Further Information

Guidance/Context

It is the responsibility of the data controller to ensure that the obligations and rights of the controller is made clear to the processor. This should align with what the data controller has outlined in the DPIA.

Response Type

Multiple Choice

Answer Criteria

N/A

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_PC01 or ORC_ERC_EDC_PC04 or ORC_ERC_EDC_PC05 are answered no.

Scoring Impact

Low risk applied if answered no.

Does the contract make clear that the processor must only act upon written instruction from the controller?

ORC_ERC_EDC_PC09

 Further Information

Guidance/Context

As part of the processing agreement it is important that the processor only acts upon instruction from the controller. This is to ensure data is handled and processed appropriately.

Response Type

Yes/No

Answer Criteria

N/A

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_PC01 or ORC_ERC_EDC_PC04 or ORC_ERC_EDC_PC05 is answered no.

Scoring Impact

Low risk applied if answered no.

Does the contract make clear to the processor their responsibilities in ensuring that their employees are subject to a duty of confidence?

ORC_ERC_EDC_PC10

 Further Information

Guidance/Context

A duty of confidence is when a person knows confidential information in circumstances where it would be unfair were that information to be disclosed to others. Processing agreements should make it clear that it is the processors responsibility to ensure that their employees handle data appropriately.

Response Type

Yes/No

Answer Criteria

N/A

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_PC01 or ORC_ERC_EDC_PC04 or ORC_ERC_EDC_PC05 is answered no.

Scoring Impact

Low risk applied if answered no.

Does the contract make clear to the processor their responsibilities in ensuring all appropriate safeguarding measures are in place to ensure the security of the data they are processing?

ORC_ERC_EDC_PC11

 Further Information

Guidance/Context

The processing contract should state what security measures need to be in place from the processor to ensure that is compliant with the standards expected by the data controller.

Response Type

Yes/No

Answer Criteria

N/A

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_PC01 or ORC_ERC_EDC_PC04 or ORC_ERC_EDC_PC05 is answered no.

Scoring Impact

Low risk applied if answered no.

Does the contract clearly state that the processor must assist the controller when responding to requests to exercise user rights?

ORC_ERC_EDC_PC12

 Further Information

Guidance/Context

There may be occurrences when the processor needs to action requests exercise to user rights, such as the right of restriction of processing. The processing contract should make it clear that the processor is accountable for assisting with any such requests.

Response Type

Yes/No

Answer Criteria

N/A

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_PC01 or ORC_ERC_EDC_PC04 or ORC_ERC_EDC_PC05 is answered no.

Scoring Impact

Low risk applied if answered no.

Does the contract clearly state that the processor must assist the controller in meeting their legal data protection requirements? (e.g. notifying of breaches and completing DPIAs).

ORC_ERC_EDC_PC13

 Further Information

Guidance/Context

The contract should ensure the processor is clear with their responsibilities to make sure that the data controller remains in line with what is laid out in their DPIA.

Response Type

Yes/No

Answer Criteria

N/A

Logic

DISABLED LOGIC Disabled if ORC_ERC_EDC_PC01 or ORC_ERC_EDC_PC04 or ORC_ERC_EDC_PC05 is answered no.

Scoring Impact

Low risk applied if answered no.

 

CLINICAL EVIDENCE

The Evidence Standards for Digital Health Technologies Framework (“ESF”) was created by the UK’s National Institute for Health and Care Excellence (“NICE”). This framework clustered app’s into relevant Tiers and identified for each Tier what forms of ‘evidence’ or ‘assurance’ would be required. It is therefore better to think of the ESF as an Assurance Standards Framework, with evidence being just one of many elements within that digital assurance matrix.

An adapted version of the ESF has been developed over time with and has now been adopted in numerous other national and pan-national Digital Health Assessment Frameworks in areas like New Zealand, Canada, Israel and the Netherlands. We conduct an analysis of any evidence available through the Review Resources. If this exists, the app is evaluated against a series of questions to determine the quality of this evidence. We look for:

·         a suitable sample size and make up;

·         a p value of below 0.05 to indicate significance;

·         a p value below 0.02 for near significance; and

·         an appropriate comparator.

This is scaled against the NICE Evidence Standards Framework and we look for a higher level of evidence for apps with more complex functionality and higher risk.

 

Question

Question Reference Source

EE02: What type/s of evidence is available?

Survey, RCT, Pilot study, Observational (Case study, Cross-sectional, Cohort), Meta-Analysis/Systematic Review

ORC_EE02

 Further Information

Guidance/Context

The purpose of this question is to identify the evidence that is available. Varying levels of evidence are required to pass the designated ESF tier. Choose all applicable from: survey, RCT, pilot study, observational study (including case study, cross-sectional or cohort), meta-analysis/systematic review, or indicated user acceptance/benefit. The follow on questions will be answerable for each evidence type chosen.

Response Type

Multiple Choice

Answer Criteria

Survey: If the app has gotten information from current users on their outcomes or how they utilise the app, and provides a description of the outcomes.

Randomised Control Trial: The research paper will state this. An RCT has two (or more) groups of people, where the only major difference should be the treatment they receive, and as the name suggests, people should be randomly assigned to these groups.

Pilot Study: A smaller-scale, preliminary study which is completed first to determine whether a study is feasible.

A Case study with no p value would come under Pilot.

Observational Study: An experimental or quasi-experimental study which demonstrates relevant outcomes. For example, a cohort study of individuals using a depression app. They measure depression before first use, and depression after eight weeks, and compare to see if there is any effect. This type of study also includes cross-sectional studies, which provide an image of people at a certain point in time. For example, it may be that people suffering pre-diabetes use an app. At the point in time studied, they have not developed type 2 diabetes, so the app may have helped.

A Case Study would come under Observational Study, if it has a p value

A cross-sectional study would come under Obs Study

Meta-analysis/Systematic Review: A systematic review refers to the entire process of selecting, evaluating, and collating all available evidence, while the term meta-analysis refers to the statistical approach to combining the data derived from a systematic-review. For our review, this may be that the evidence provided has pulled together all the studies about an app to provide a single p-value to demonstrate the app’s significance.

Indicated user acceptance/benefit: A statement or other piece of information which indicates a benefit of the app to users, or indication that the app has undergone a pilot study. This option is to be selected when you are unable to see further evidence that supports any claimed facts or outcomes. For example, the developer website states “9/10 users found their sleep improved”, but you can’t see the evidence behind this statement. Testimonials on the website can be accepted, but not from the app store review section. Any statement of users benefiting from the app.

Logic

There is no disablement logic written for this question.

Scoring Impact

Varying amounts of risk or value applied depending on the defined ESF tier of the app. Higher tiered apps require more substantial evidence (i.e. an RCT study), if this is not identifiable and the app fails other criteria to pass at it’s defined ESF tier then more risk is applied. If the evidence is appropriate to the designated ESF tier and the app meets other criteria to pass it’s ESF tier then value is applied.

How many pieces of evidence does the app provide?

ORC_EE14

 Further Information

Guidance/Context

This is a checkbox question, with the choice to select the amount of evidence found. For each piece of evidence up to 5 max, the follow on questions will need to be completed. Anymore than 5, the best 5 pieces should be chosen.

Response Type

Multiple Option

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

How many RCT's and/or observational studies does the app have?

ORC_EE13

 Further Information

Guidance/Context

This is a checkbox question, with the choice to select the amount of evidence found. For each piece of evidence up to 5 max, the follow on questions will need to be completed. Anymore than 5, the best 5 pieces should be chosen.

Response Type

Multiple Option

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

For each type of relevant evidence:

EE10: What category does the evidence relate to?

ORC_EE10 

 Further Information

Guidance/Context

This is about the ORCHA category to which the app relates. It may be there is evidence for more than one category, if this is the case input all that apply.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

For each type of relevant evidence:

EE11: What benefit does the evidence relate to?

ORC_EE11          

 Further Information

Guidance/Context

This is about the ORCHA benefit to which the app relates. It may be there is evidence for more than one benefit.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

EE03: Provide links to the publicly available evidence/published evidence that the developer has provided.

ORC_EE03

 Further Information

Guidance/Context

This is a free text option, the answer should contain the links to the evidence found. Only place one link in the text box for each of these.

Response Type

Free Text

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

For each type of relevant evidence:

EE04: Is the sample size appropriate?

ORC_EE04          

 Further Information

Guidance/Context

This is about the sample size of any type of RCT or observational study that has been identified. There are no scoring implications as this is for data collection only.

Response Type

Yes/No

Answer Criteria

Yes: If the sample size is equal to or above 30 participants.

No: If the sample size is below 30 participants OR the sample size is not mentioned.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

For each type of relevant evidence:

EE05: Does the evidence found provide a p-value?

ORC_EE05

 Further Information

Guidance/Context

N/A

Response Type

Yes/No

Answer Criteria

Yes: If the research paper/article provides a p-value/confidence interval (CI). This will likely be found within the abstract and/or results section.
Confidence interval example - (95% CI: 6.4–7.2).

No: If the research paper/article does not provide a p-value.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

For each type of relevant evidence:

EE06: Does the p-value demonstrate significance (p<0.05)?

ORC_EE06

 Further Information

Guidance/Context

This question is used to provide indication on whether or not the research article has proven a benefit. There are of course other ways to do this, but the use of a p-value is the most common one. Other situations should be considered on a case by case basis.

Response Type

Yes/No

Answer Criteria

Yes: If the p-value identified is a number less than or equal to 0.05.

No: If the p-value identified is a number greater than 0.05.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

For each type of relevant evidence:

EE12: Does the p-value demonstrate near significance (p<0.2)?

ORC_EE12

 Further Information

Guidance/Context

ORCHA uses p-values to see if an app has demonstrated a benefit. It is possible that the app can have a benefit, but for the purposes of this one particular study it has not reached the accepted significance level, but do come close. ORCHA use this question to recognise this.

Response Type

Yes/No

Answer Criteria

Yes: If the p-value identified is a number less than or equal to 0.2.

No: If the p-value identified is a number greater than 0.2.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

For each type of relevant evidence:

EE07: Is there a comparator?

ORC_EE07

 Further Information

Guidance/Context

The use of a comparator allows a comparison to be made between the app’s benefits and something else. This gives context for the benefits that the app may or may not have demonstrated

Response Type

Yes/No

Answer Criteria

Yes: If the research article/paper identified has a separate group than that which is the experimental condition. For example if the researcher is comparing against a baseline which may have come from the user prior to the intervention.

The comparator could be as simple as paid version of app vs free version of an app.

No: If the research article/paper identified has only the experimental condition.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

For each type of relevant evidence:

EE08: Is the comparator validated?

ORC_EE08

 Further Information

Guidance/Context

The use of a comparator allows a comparison to be made between the app’s benefits and something else. This gives context for the benefits that the app may or may not have demonstrated. A validated comparator means a current standard treatment pathway. An example may be a depression app being compared to an antidepressant.

Response Type

Yes/No

Answer Criteria

Yes: If the research article/paper identified has a separate group than that which is the experimental condition, and the statement about that group includes “Current standard of care, Usual care or Treatment as usual”.

No: If the research article/paper identified has only the experimental condition, or the paper does not mention “current standard of care, usual care or treatment as usual”.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

 

Behavioural Change

There are some scenarios where the app utilises widely accepted techniques with a breadth of evidence. In this instance the app may not deem it appropriate to fund a full randomised control trial to demonstrate effectiveness. Therefore we give some value points for fully referencing evidence for behavioural change techniques used within the app. This is not however treated in the same way as where the app has provided direct evidence of its own effectiveness.

Question

Question Reference Source

Does the App have its own high quality study?

ORC_BCT01  

 Further Information

Guidance/Context

The purpose of this question is to identify evidence that the app has performed its own study on behaviour change techniques, which meets the ESF requirements of the tier. This is information gathering, and is more important for the following question, BCT02.

Response Type

Yes/No

Answer Criteria

Yes: If the research article/paper identified is suitable evidence for an app of that ESF tier. For example, if the app is tier 3b on the NICE ESF, and the evidence identified is an RCT, with a significant p value and validated comparator.

Tier 3a - Needs to be minimum of an observational study with a comparator and a significant P value/confidence interval.

Tier 3b - needs to be an RCT with significant P Value/Confidence Interval and validated comparator.

Any tier below 3a, the app would only need to tick off the 3a requirements to answer this question yes.

No: If the research article/paper identified does not have a high quality study, suitable for an app at that tier. For example, the app is tier 3b on the NICE ESF, but the evidence identified is an observational study but ongoing

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the App reference and evidence its behaviour change technique?

ORC_BCT02

 Further Information

Guidance/Context

This question is to differentiate those apps which don’t have a study. If the app has not got its own study and they use a behaviour change technique, then this question looks to see if it is referenced. If it is referenced then this allows a small increase in value to the scoring, even though the developer themselves may not have the necessary study demonstrating efficacy of the specific product.

Response Type

Yes/No

Answer Criteria

YES: If the developer displays research on which the app is based. For example, an app may have built a feature into their app based on other research, or they may refer to a paper about the psychological intervention it is based on, eg CBT, “we added these features based on this paper” etc.

No: If the developer does not reference or evidence the behaviour change the app is based on OR if they mention briefly, but don’t provide specific links.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

 

Professional Backing

We look for evidence of an appropriate professional being involved in the app's design and development, or if the app has been externally accredited. A relevant professional is deduced in the context of an app. For example, for a simple yoga app we would accept a qualified yoga instructor as a relevant professional, but for a complex clinical solution we would only accept a relevantly qualified clinician. External accreditations are wide ranging, but we would look for an appropriate body, for example the British Heart Foundation giving an endorsement to a cardiology app.

Question

Question Reference Source

Is there a suitably qualified Professional involved in the development team of the App?

ORC_PB01

 Further Information

Guidance/Context

This question looks to identify if there was a relevant professional part of developing the app, this helps indicate that the information contained within is relevant.

Response Type

Yes/No

Answer Criteria

YES: If there is evidence of a suitably qualified professional being involved with the app. For example, a CBT website displays a psychologist on the “Our team” page of their website.
NB It is important to attempt validate this named professional.

No: If the developer does not reference a suitably qualified professional OR it is not clear what role they play. For example, a psychologist is named, but it is unclear whether they simply use the app, or where involved in the development.

Logic

There is no disablement logic written for this question.

Scoring Impact

Varying value based on ESF tier of app. Value is not awarded for both PB01 and PB02.

Does the organisation behind the App have relevant credentials? 

ORC_PB02

 Further Information

Guidance/Context

This question generally looks to assist larger organisations that may not have the ability or practicality to name individuals involved in the creation of the app.

Response Type

Yes/No

Answer Criteria

YES: If the app is made by an institution that is believed to have the relevant experience. For example, the app is produced by the NHS, or Public Health England.

No: If the app does not have any relevant credentials, and was simply produced by a development company.

Logic

There is no disablement logic written for this question.

Scoring Impact

Varying value based on ESF tier of app. Value is not awarded for both PB01 and PB02.

Is there evidence of an endorsement by a relevant body?

ORC_PB03

 Further Information

Guidance/Context

This question helps provide assurance that the app works as intended from any endorsements that can be seen from relevant healthcare bodies.

Response Type

Yes/No

Answer Criteria

YES: If there is evidence of any sort of accreditation by any sort of relevant organisation, or professional body. For example, a Diabetes app with an endorsement from the National Diabetes Foundation. (CE marks that are HIGHER than a Class 1, MHRA, NICE and FDA approval count here). NHS apps would also get a yes, and those on the NHS Apps library.

ISO 13485 counts towards an external endorsement.

No: If the app does not have any relevant endorsements OR if the endorsement is from an individual, rather than an organisation or body OR if the organisation endorsing the app are in some way involved with the development/content of the app.

Logic

There is no disablement logic written for this question.

Scoring Impact

High value if answered yes.

Are organisations using the App?

ORC_PB04

 Further Information

Guidance/Context

This question helps provide assurance that the app works as intended if healthcare organisations have adopted the app and are using the app.

Response Type

Yes/No

Answer Criteria

YES: If there is evidence (can be a statement) of any sort of relevant organisation using the app. For example, the website may display that the platform is used by a CCG, or display the relevant logos.

No: If it is not clear any organisations are using the app OR if a person references their position in an organisation, but doesn’t make it clear it is organisational use.

Logic

There is no disablement logic written for this question.

Scoring Impact

Varying value if yes based on ESF tier of app.

Is there a statement that it has been positively evaluated or validated by a relevant healthcare professional?

ORC_PB05

 Further Information

Guidance/Context

This question helps provide assurance that if a healthcare professional is willing to positively evaluate an app, using their own name/qualifications, it provides assurance that the app works as expected.

Response Type

Yes/No

Answer Criteria

ES: If there is evidence of any sort of testimonial or accreditation by any sort of relevant individual (external from company). For example, a Diabetes app accredited by a Diabetologist.

No: If the app does not have any relevant endorsements OR if the endorsement is from an organisation or body, rather than an individual.

Logic

There is no disablement logic written for this question.

Scoring Impact

Varying value based on ESF tier of app.

Please specify who the relevant experts are and what qualifications they hold.   

ORC_AE17·          

 Further Information

Guidance/Context

For data collection purposes, please record who the relevant expert is. Where possible the qualifications of the professional should be validated.

Response Type

Free text

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is there evidence within the app that the developer has validated any Guidance/Context with relevant reliable information sources or references? 

ORC_PB06

 Further Information

Guidance/Context

The point of this question is to establish whether the information provided comes from a relevant and reliable source (can be a qualified person/organisation/citation of original journal article).

Response Type

Yes/No

Answer Criteria

YES: If a link to a source is provided - even if the link can’t be clicked/pressure, if it can be typed it in and it’s valid then yes OR if the developer uses a well established tool, which they reference (GAD-7, PHQ-9 etc) OR if the developer links to external information, which comes from a reputable outside source (NHS choices, PHE etc.).

No: If the app does not have any relevant guidance/context which has been validated, either in the form of references, or using cleared named clinical calculators.

Logic

There is no disablement logic written for this question.

Scoring Impact

Varying value based on ESF tier of app.

ORCHA Adapted ESF Compliance

The first part of this section assesses which ESF Tier the app falls under, and is non-scoring. The second part assesses whether the app meets the minimum requirements of that Tier. Compliance with the ESF is determined by the app answering positively to all questions that have been flagged as a requirement for its Tier of the ESF and all Tiers below.

 

Question

Question Reference Source

What Tier of the ESF is the App?

 

Is the app Tier 1?

ORC_ESF01

 Further Information

Guidance/Context

Classification for tiering of an app is defined by the scene setter questions asked to help assess the functions and features of an app. To be classified as a tier 1 app it must-

Have met none of the requirements for any other tier & provides no patient outcomes.

Example: maintenance app OR admin app- Acts as a healthcare system/service.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is the app Tier 2a?

ORC_ESF02

 Further Information

Guidance/Context

Classification for tiering of an app is defined by the scene setter questions asked to help assess the functions and features of an app. To be classified as a tier 2a app it must-

Provide information or guidance/context (I01 is yes).
OR
Allow a healthcare professional to provide clinical advice, as opposed to the app providing it (EF09 is yes).
OR
Provide information, resources, or activities to the public, patients, or clinicians, either about a specific condition or general health and lifestyle (EF07 is yes).
OR
Provide two-way communication between patients, citizens or healthcare professionals (EF10 is yes).
OR
The app is a simple self-management app (selected in MN04).

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is the app Tier 2b?

ORC_ESF03

 Further Information

Guidance/Context

Classification for tiering of an app is defined by the scene setter questions asked to help assess the functions and features of an app. To be classified as a tier 2b app it must-

Do none of the things listed in 3a/3b & it is a standard self management app as defined by the scene setter questions (MN04).

Example: If an app is simple monitoring with a specific condition focus OR complex monitoring with a wellbeing and general health focus, as defined by the scene setter questions.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is the app Tier 3a?

ORC_ESF04

 Further Information

Guidance/Context

Classification for tiering of an app is defined by the scene setter questions asked to help assess the functions and features of an app. To be classified as a Tier 3a app it must-

Have no things listed in 3b
AND
Be a complex self management app (selected in MN04).
OR
Have preventative behaviour change within the app (selected in TS11).
OR
Have a recognised (not novel) clinical calculator within the app (TS01 is yes and TS02 mentions an established clinical calculator).

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is the app Tier 3b?

ORC_ESF05

 Further Information

Guidance/Context

Classification for tiering of an app is defined by the scene setter questions asked to help assess the functions and features of an app. To be classified as a Tier 3b app it must-

- Diagnose a condition (DG02 is yes).
OR
- Have a novel clinical calculator within the app which impacts care, treatment, or diagnosis (TS01 is yes and TS02 mentions a novel clinical calculator).
OR
- Automatically measures and/or records data about a user’s specified condition, and transmits the data to a professional, carer, or third party organisation, without any input from the user (MN07 is yes).
OR
- Provide treatment (TS05 is yes).
OR
- Guide the treatment of a condition (TS07 is yes).
OR
- Alleviate the symptoms of an existing condition (TS15 is yes).

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the App met:

 

Tier 1 minimum requirements?

ORC_ESF06

 Further Information

Guidance/Context

The app has met Tier 1 requirements if the app has:

-Evidence of a survey, pilot study, meta-analysis, RCT, observational, or other indicated user acceptance/benefit (EE02 does not contain none).
AND at least one of the following has been answered yes:
- Evidence of a relevant professional involved in the development team (PB01).
- Relevant organisational credentials (PB02).
- Evidence of endorsement by a relevant body (PB03).

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

High risk if no or high value if yes + multiplier for value applied based on evidence available.

Tier 2a requirements?

ORC_ESF07

 Further Information

Guidance/Context

The app has met tier 2a requirements if the app has all of the following criteria:

- Evidence that the developer has validated the information, advice or Guidance/Context with relevant academic expert input (PB01 or PB02 or PB06 is yes).
AND
- There is clear evidence of safeguarding measures being in place for any communication functions (AE13 is yes, if applicable).
AND
- The app has evidence of accrediting expertise (PB01 or PB02 or PB03 or PB05 is yes).

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

High risk if no or high value if yes + multiplier for value applied based on evidence available.

Tier 2b requirements?

ORC_ESF08

 Further Information

Guidance/Context

The app has met tier 2b minimum requirements if the app has:

- Evidence that the developer has validated the information, advice or Guidance/Context (PB01 or PB06 is yes).
AND
- Clear evidence of safeguarding measures being in place for any communication functions (AE13 is yes, if applicable).
AND
- Evidence of accrediting expertise (PB01 or PB02 or PB05 is yes).
AND
- Evidence of an endorsement by a relevant body (PB03 is yes).
OR
- A meta-analysis, or an observational study/RCT with a p-value < 0.05 (EE02 is a yes AND one of the EE06 answers is a yes).

NB - If an app has met tier 1, tier 2a AND tier 3a requirements, then the app will have met 2b requirements.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

High risk if no or high value if yes + multiplier for value applied based on evidence available.

Tier 3a requirements?

ORC_ESF09

 Further Information

Guidance/Context

The app has met Tier 3a minimum requirements if it has:

- Evidence of an RCT (EE02 answer includes RCT) which has a significant p value (EE06 is yes).
OR
- Evidence of an observational study (EE02 answer includes observational) which has a significant p value (EE06 is yes).
AND
- A comparator (EE07 is yes) OR a validated comparator (EE08 is yes).

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

High risk if no or high value if yes + multiplier for value applied based on evidence available.

Tier 3b requirements?

ORC_ESF10

 Further Information

Guidance/Context

The app has met Tier 3b minimum requirements if it has:

- Evidence of an RCT (EE02 answer includes RCT) which has a significant p value (EE06 is yes).
AND
- A validated comparator (EE08 is yes).

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

High risk if no or high value if yes + multiplier for value applied based on evidence available.

Does the app have appropriate evidence for the ESF tier?

ORC_ESF11

 Further Information

Guidance/Context

Use the above questions as a guide to determine the answer. YES if the app has met its own tier, plus those below, as the requirements are cumulative. If the app has met only the requirements at its own tier and not those below, then this should be answered NO.

If no, provide an explanation why.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Medical Devices

It is proposed that the NORDIC Assessment assess if the app is likely to be a medical device under the current Guidance/Context from the MDR (https://ec.europa.eu/growth/sectors/medical-devices_en ). We then evaluate if the app displays the relevant CE mark.

Question

Question Reference Source

Is the app a medical device?

ORC_MD11

 Further Information

Guidance/Context

N/A
It is important to define if an app is classified or could possibly be classified as a medical device due to the necessary standards they would need to meet.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the app have a CE mark?    

ORC_AE06

 Further Information

Guidance/Context

N/A

Response Type

Yes/No

Answer Criteria

YES: If the app or accompanying website displays a CE mark specific for the app.

NO: If the app does not have a CE mark.

If the app has been assessed by the MHRA to not require a CE mark.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the app state that it has been assessed by the MHRA or other relevant body, and does not require a CE mark? 

ORC_AE08

 Further Information

Guidance/Context

N/A

Response Type

Yes/No

Answer Criteria

YES: If the app has been assessed by the MHRA, Amtac Certification Services Ltd (0473), BSI Healthcare (0086), Lloyd’s Register Quality Assurance Ltd (0088). SGS United Kingdom Ltd (0120), UL International (UK) Ltd (0843) - and deemed to NOT require a CE mark.

NO: If the app has a CE mark.

If the app has been assessed by the MHRA, but NOT been declared to not require a CE mark.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

What class is the app certified as?           

ORC_MD09

 Further Information

Guidance/Context

Medical device comes at different levels of certification. If the app is anything higher than a class 1, then the regulatory code will be displayed next to the CE mark. This can be found within the app or on the developer website.

A CE mark on its own = Class 1

A CE mark with series of numbers after it = Class 2

Response Type

Free Text

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the app been FDA approved? (Food and Drug Administration)  

ORC_FDA01

 Further Information

Guidance/Context

The question is looking for if the FDA has approved a premarket approval (PMA) application, or a Humanitarian Device Exemption (HDE) application. This is for class III medical devices (highest risk) and involves a more rigorous review than the 510(k) review process.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the app been FDA cleared?             

ORC_FDA02

 Further Information

Guidance/Context

FDA CLEARANCE means that an app uses a feature/algorithm which itself has been FDA approved, and the app has been cleared to use the same feature which functions as it should do.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Clinical Safety/Risk Management

It is proposed that the evaluation looks for any safeguarding measures in communication functions of the app, if relevant.

Question

Question Reference Source

Is there a statement or any evidence showing that appropriate safeguarding measures are in place around peer-support and other communication functions within the platform?

 

ORC_AE13

 Further Information

Guidance/Context

This question is a Tier 2a and above requirement. It is only asked of apps that require such measures because of the functional capabilities/intended purpose of the app.

Response Type

Yes/No

Answer Criteria

YES: If there is an internal forum, the content is moderated, or guidelines are set out, or it is monitored OR there is a full policy in place specific to a forum OR if two-way communication occurs and the data is protected/encrypted OR if there is a registration process where you use a HCP number OR if the communication is made clear it is only with a registered HCP.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the Developer clearly identify who the app should and should not be used by? 

ORC_S01

 Further Information

Guidance/Context

The question is looking for a statement or other evidence showing who the app is intended for. Or if there are any demographics who should not use it.

Response Type

Yes/No

Answer Criteria

YES: If a developer tells us who the app SHOULD or SHOULD not be used by. Can be specific or general e.g. for 18years +, for anyone who undertakes physical activity, etc.

No: If the app does not tell us who the app SHOULD or SHOULD not be used by.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the Developer publish their risk management processes?

ORC_S02

 Further Information

Guidance/Context

It is understood that risk management documents may contain company sensitive information. Therefore documents do not have to be made publicly available but could be made available upon request, or there could be a detailed explanation for the process involved within the developers risk management process.

Response Type

Yes/No

Answer Criteria

YES: If there is evidence of a risk management process. This may be in the form of a hazard log or safety case, and will likely be made available through the website, if available at all.

No: If the developer does not clearly display their risk management processes.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the Developer make clear risks associated with using the app?

ORC_S03

 Further Information

Guidance/Context

This question provides context to the user to make an informed decision about the risks associated with the app, and whether a user would still want to use it.

Response Type

Yes/No

Answer Criteria

YES: If the developer defines clearly what possible risks there are to a user - this may be in the form of a hazard log or safety case. A disclaimer highlighting the risks. 

No: If the developer does not clearly display the risks associated with their app.

A disclaimer highlighting that information in the app is not medical advice or something along those lines is not suitable to meet these requirements. 

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is there a way for the user to confirm that the data input is accurate?

ORC_S04

 Further Information

Guidance/Context

This question is looking to identify what validation is applied to data inputted by the users. It is looking to see if the app checks for erroneous data, this helps ensure the safety of the app by not allowing mis-calculations from inputted data.

Response Type

Yes/No

Answer Criteria

YES: If data is being entered the app requires confirmation. For example if a users was to input 5000 mmol/l for glucose readings, does it ask for confirmation? If the app uses sliders to restrict data entry parameters then this would also be a yes. Should be noted that this should somehow be related to the app function, rather than other data input.

No: If the developer does not ask the user to confirm input.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the developer list a Clinical Safety Officer (CSO) on/in any relevant sites/content?

ORC_S05

 Further Information

Guidance/Context

A CSO is a person responsible for the management of the clinical risk processes ensuring that documentation is accurate and that any processes are being followed correctly.

Response Type

Yes/No

Answer Criteria

YES: If there is evidence of a named Clinical Safety Officer, anywhere within the app or associated sites/documentation. The role should be named as a CSO, and cannot accept just a Doctor, or Psychologist etc.

No: If the developer does not clearly name their CSO.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Please provide more detail.

ORC_S06

 Further Information

Guidance/Context

The purpose of this question is to collect the details about the relevant CSO.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is the App/Solution in scope for a clinical safety assessment?

ORC_ERC_OCSA_CSS1

 Further Information

Guidance/Context

The purpose of this question is to define if the app is in scope and to determine the need for the following questions.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Why is the app in scope?

ORC_ERC_OCSA_CSS2

 Further Information

Guidance/Context

To ensure understanding it is needed to define why the app is in scope of needing a clinical safety assessment.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the Developer provided a thorough summary about why the App is out of scope? (Eg is it complete? Does it match the functionality put forward?)

ORC_ERC_OCSA_CSS3

 Further Information

Guidance/Context

If the Developer suggests the app is out of scope, and they believe it is unlikely to cause clinical risk, they need to have an appropriate explanation. This question allows assessors to check if it is thorough.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

What risks if any, have been documented concerning harm to a patient?

ORC_ERC_OCSA_CSS4

 Further Information

Guidance/Context

This is to ensure harms are documented, and it ensures the end user considers them in the context of the patient, rather than anything else, as required.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

If the App/Solution has been deemed in scope, has the developer supplied suitable Risk Management Documentation?

ORC_ERC_OCSA_CSS5

 Further Information

Guidance/Context

The developer should provide all documentation that should be included with in the risk management documentation.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Please confirm the name of your Clinical Safety Officer (CSO), their profession and registration details?

ORC_DTAC_CS120

 Further Information

Guidance/Context

The developer should provide details of their named CSO.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is the CSO a suitably qualified and experienced clinician?

ORC_ERC_OCSA_CSO1

 Further Information

Guidance/Context

It is important that the CSO has suitable qualification related to the product. If this is not detailed in the documentation, research may be needed on the company website, or LinkedIn to find a suitable individual.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the named CSO have appropriate qualifications and up to date registration details?

ORC_DTAC_CS121

 Further Information

Guidance/Context

The CSO needs to have a current registration, so a retired GP would not be accepted. This is to ensure they are up to date with best practices.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is there evidence the CSO has played an active part in the clinical safety process – approval of the risk management file, hazard assessment participation etc?

ORC_ERC_OCSA_CSO3

 Further Information

Guidance/Context

The CSO should be listed as having authored, reviewed or approved the documentation. The summary safety statement must be signed off by the CSO.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the developer provided their Clinical Safety Case and Hazard Log?

ORC_DTAC_CS112

 Further Information

Guidance/Context

To pass, the developer is required to submit the Clinical Safety Case Report and Hazard Log that is compliant with the requirements set out in DCB0129. This should be commensurate with the scale and clinical functionality of the product and address the clinical risk management activities specified with the standard.

The Clinical Safety Care Report should present the arguments and supporting evidence that provides a compelling, comprehensible and valid case that a system is safe for a given application in a given environment at the defined point in the products lifecycle.

It should provide the reader with a summary of all the relevant knowledge that has been acquired relating to the clinical risks associated with the product at that point in the lifecycle including:

- A clear and concise record of the process that has been applied to determine the clinical safety of the product;
- A summary of the outcomes of the assessment procedures applied;
- A clear listing of any residual clinical risks that have been identified and the related operational constraints and limitations that -are applicable;
- A clear listing of any hazards and associated clinical risks that have been transferred, together with any declared risk control measures, that are to be addressed as part of the clinical risk management process in the organisation where the product is being deployed;
- A listing of outstanding test issues / defects associated with the product which may have a clinical safety impact.

The Hazard Log should record and communicate the on-going identification and resolution of hazards associated with the product. All foreseeable hazards should be identified and the risk of such hazards should be reduced to acceptable levels.

A summary should also be provided to the assessor of identified hazards that the developer has been unable to mitigate to as low as it is reasonably practicable. It should also clearly identify the hazards which will require user or commissioner action to reach acceptable mitigation.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the Developer outline the need for the Risk Management Documentation?

ORC_ERC_OCSA_CSD5

 Further Information

Guidance/Context

Does the developer understand why they have had to create and submit this documentation. This should be made clear in the documentation that they provide.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does it have full version history and issue date published?

ORC_ERC_OCSA_CSD2

 Further Information

Guidance/Context

This question is included to allow us to check that the developer has appropriate risk management documentation, and a robust clinical safety process in place. Version history and issue date should be correct and up to date, beginning with version 1.0.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the Developer described their clinical risk management system? ( identification of key personnel, their roles and responsibilities; identification of clinical risk management governance structure.

ORC_ERC_OCSA_CSD6

 Further Information

Guidance/Context

This question is included to allow us to check that the Developer has appropriate risk management documentation, and a robust clinical safety process in place. The Safety Case should include all the necessary headings and information - An NHS template can be used to help Developers do this correctly.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the Safety Case make a mention of a test summary? (Summary of any outstanding test issues and the impact on clinical safety)

ORC_ERC_OCSA_CSD7

 Further Information

Guidance/Context

This question is included to allow us to check that the Developer has appropriate risk management documentation, and a robust clinical safety process in place. Any outstanding issues should be summarised - or the safety case should state that there are no outstanding issues.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the Safety Case have a summary statement showing sign off from the CSO?

ORC_ERC_OCSA_CSD8

 Further Information

Guidance/Context

This question is included to allow us to check that the Developer has appropriate risk management documentation, and a robust clinical safety process in place. The summary statement should be signed off by CSO.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is there evidence of a CSO reviewing, contributing or approving the Safety Case?

ORC_ERC_OCSA_CSD3

 Further Information

Guidance/Context

This question is included to allow us to check that the Developer has appropriate risk management documentation, and a robust clinical safety process in place. The CSO should be listed as having authored, reviewed or approved the Safety Case.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the Hazard Log have full version history and issue date published?

ORC_ERC_OCSA_CSD9

 Further Information

Guidance/Context

This question is included to allow us to check that the Developer has appropriate risk management documentation, and a robust clinical safety process in place. The log should not have any missing dates or names. It is important to ensure that the hazard log has been updated as regularly as the process document suggests.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Are the hazards listed complete? (From a review of the listed hazards, do they have all of the details required completed such as name, clinical impact and risk ratings. Do the risk ratings look appropriate or do they appear to be copy and paste throughout the listed hazards? Are they scored on the low side? Does the consequence change pre and post assessment?

ORC_ERC_OCSA_CSD4

 Further Information

Guidance/Context

This question is included to allow us to check that the Developer has appropriate risk management documentation, and a robust clinical safety process in place. Ensure that all hazards have been filled out correctly. Confirmation on if there may be errors or unconsidered hazards can be discussed with ORCHA’s CSO.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Are the potential harms related to the user/patient?

ORC_ERC_OCSA_CSD11

 Further Information

Guidance/Context

This question is included to allow us to check that the Developer has appropriate risk management documentation, and a robust clinical safety process in place. Within the documentation, identified risks must directed at the perspective of the user/patient.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Do the harms outline what the clinical impact may be for the user?

ORC_ERC_OCSA_CSD12

 Further Information

Guidance/Context

As with the penultimate questions, clinical risk should always be considered from the end user’s perspective. This question reinforces the view, and ensures that the developer is considering clinical impacts from the user’s perspective.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the Developer covered all of the possible causes for each Hazard?

ORC_ERC_OCSA_CSD13

 Further Information

Guidance/Context

It is possible for hazards to have more than one cause, this question is about making sure the app developer has considered all of the causes for a possible hazard, and not just one. This means they can correctly mitigate.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Do the risk ratings look appropriate or do they appear to be copy and paste throughout the listed hazards?

ORC_ERC_OCSA_CSD14

 Further Information

Guidance/Context

The risk ratings should match when compared to the risk matrix provided by the developer. Risks should also be checked to ensure that they are accurate/realistic. This means that if there is a significant risk of injury it should be reflected appropriately in the scoring.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Are hazards split incorrectly into potential and actual harm?

ORC_ERC_OCSA_CSD15

 Further Information

Guidance/Context

Risks should not be split into potential and actual harm - all risks should be potential, rather than hazards which have happened. If they are split, seek further advice.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is there evidence of a CSO reviewing, contributing or approving the hazard log?

ORC_ERC_OCSA_CSD10

 Further Information

Guidance/Context

This question is included to allow us to check that the Developer has appropriate risk management documentation, and a robust clinical safety process in place. The CSO should be listed on the cover page.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the Developer implemented the clinical risk analysis activities defined in the Clinical Risk Management Plan?

ORC_ERC_OCSA_CRAP1

 Further Information

Guidance/Context

Check that the risk analysis has been carried out as described - if any of the risk controls can be validated, these should be checked. The risk scores should match up to the output suggested by the risk matrix.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is the Clinical Risk Analysis carried out by a multi-disciplinary group?

ORC_ERC_OCSA_CRAP2

 Further Information

Guidance/Context

In this context multi-disciplinary means 2 or more people being involved who have a variation of experience/qualification/job title.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the Developer defined the clinical scope of the Health IT System which is to be delivered?

ORC_ERC_OCSA_CRAP3

 Further Information

Guidance/Context

This question is included to allow us to check that the Developer has appropriate risk management documentation, and a robust clinical safety process in place. Does the Safety Case describe an overview of the product and detail why it has been developed?

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the Developer defined the intended use of the Health IT System which is to be delivered?

ORC_ERC_OCSA_CRAP4

 Further Information

Guidance/Context

This question is looking to identify if the Safety Case describes how the product is intended to be used, and by who?

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the Risk Management Documentation make it clear where the App fits into the Clinical Workflow – How would a patient use the app appropriately to become well again?

ORC_ERC_OCSA_CRAP5

 Further Information

Guidance/Context

This question is looking to identify if the Safety Case describes in what setting the product should be used, or at what point in a clinical pathway?

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the Developer outlined Third Party Products integrated within the Health IT System to be released?

ORC_ERC_OCSA_DSP1

 Further Information

Guidance/Context

This question is included to allow us to check that the Developer has appropriate risk management documentation, and a robust clinical safety process in place. If the product integrates with any third party apps or devices this should be listed in the Safety Case.

Response Type

Multiple Option

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the Developer deploying the Health IT System considered how it will impact on the current business processes and ways of working?

ORC_ERC_OCSA_DSP2

 Further Information

Guidance/Context

This is question is looking to identify if the product is replacing a system or process already in place, eg. digital health checks to replace regular health checks.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is there usability and human factors related evidence within the scope?

ORC_ERC_OCSA_DSP3

 Further Information

Guidance/Context

Does the Safety Case describe anyone involved in the safety documentation as being a user, or representing a user interest group? User testing to identify risks is sufficient here. 

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the Developer assessed any infrastructure at the Health Organisation that is within their scope of influence, required to support the deployment of the Health IT System? (This may be achieved by the Manufacturer specifying the minimum system requirements)

ORC_ERC_OCSA_DSP4

 Further Information

Guidance/Context

If the product requires any specific technology or requirements to use, this should be described. E.g. software or devices required, or does it only work on the latest apple/android update?

Response Type

Multiple Option

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Where data migration is to be undertaken by the Developer it should be included in the scope of the clinical risk management activities.
Is the Developer undertaking any Data Migration?

ORC_ERC_OCSA_DSP5

 Further Information

Guidance/Context

This question is looking to capture if data migration takes place to help identify the need for follow on questions about the appropriateness and robustness of the risk management documentation, and their clinical safety process.

Response Type

Yes/No

Answer Criteria

Yes: If data is transferred away from the app.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is the Data Migration being undertaken by the Developer properly covered in the documentation?

ORC_ERC_OCSA_DSP6

 Further Information

Guidance/Context

Data risks should be considered, and data should be protected. To answer yes, hazards associated with the data migration should be analysed and suitably mitigated, working in conjunction with the relevant Health Organisation as appropriate.

Response Type

Yes/No

Answer Criteria

Yes: Hazards associated with the data migration should be analysed and suitably mitigated, working in conjunction with the relevant Health Organisation as appropriate.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the Developer identified any hazards associated with the data migration, analysed and suitably mitigated? (working in conjunction with the relevant Health Organisation as appropriate)

ORC_ERC_OCSA_IPH1

 Further Information

Guidance/Context

Data risks should be considered, and data should be protected. To answer yes, hazards associated with the data migration should be analysed and suitably mitigated, working in conjunction with the relevant Health Organisation as appropriate.

Response Type

Yes/No

Answer Criteria

Yes: Hazards associated with the data migration should be analysed and suitably mitigated, working in conjunction with the relevant Health Organisation as appropriate.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the Developer considered the end to end clinical process, including functionality and how that functionality is used?

ORC_ERC_OCSA_IPH2

 Further Information

Guidance/Context

The risks on the hazard log should be checked, to ensure that the full user journey has been taken into account, including all features of the app.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the Developer considered inter and intra Health IT System messaging?

ORC_ERC_OCSA_IPH3

 Further Information

Guidance/Context

This is only applicable if there is any internal or external messaging - then risks here should be considered about this messaging.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the Developer assessed the health IT system architecture and design?

ORC_ERC_OCSA_IPH4

 Further Information

Guidance/Context

This question is looking to identify any risks relating to how things move around the app, how the app is built, eg. if there are two databases.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is there a clear matrix, which is used to define the risk ratings?

ORC_ERC_OCSA_CRE11

 Further Information

Guidance/Context

A clear identifiable risk matrix is needed to understand the risk ratings.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

For each identified hazard, has the Developer evaluated whether the initial clinical risk is acceptable?

ORC_ERC_OCSA_CRE9

 Further Information

Guidance/Context

Does the Hazard Log or Safety Case provide an acceptability criteria - and have the initial risks been deemed acceptable? If not - the residual risk should be acceptable.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the Developer used the risk acceptability criteria previously defined?

ORC_ERC_OCSA_CRE10

 Further Information

Guidance/Context

This question is looking to identify if the risk acceptability that is shown is based off of what was defined in the risk matrix previously provided.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the Developer identified appropriate clinical risk control measures to remove any unacceptable clinical risk?

ORC_ERC_OCSA_CRE1

 Further Information

Guidance/Context

The question looks to identify if the controls in place appear appropriate, and do they reduce the risk rating? The controls should also not introduce any additional risk.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the Developer assessed Proposed clinical risk control measures to determine whether new hazards will be introduced as a result of the measures?

ORC_ERC_OCSA_CRE2

 Further Information

Guidance/Context

If any additional risk has been introduced as a result of mitigations that have been brought in, these must be assessed.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the Developer assessed proposed clinical risk control measures to determine whether the clinical risks for previously identified hazards will be affected?

ORC_ERC_OCSA_CRE3

 Further Information

Guidance/Context

It is important to ensure that any controls in place for additional risks, must not have an impact on existing risks or controls.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is the Developer managing new hazards, or increased clinical risks?

ORC_ERC_OCSA_CRE4

 Further Information

Guidance/Context

If there are any new risks that are introduced the additional or increased risks must then be mitigated against.

Response Type

Multiple Option

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

For each identified hazard, has the Developer evaluated whether the residual clinical risk is acceptable?

ORC_ERC_OCSA_CRE5

 Further Information

Guidance/Context

Residual risk is the risk that is remaining when all mitigations and controls are in place. Residual risk must be at an acceptable level as defined by the risk matrix?

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the Developer used the risk acceptability criteria previously defined?

ORC_ERC_OCSA_CRE6

 Further Information

Guidance/Context

This question is looking to identify if the risk acceptability that is shown is based off of what was defined in the risk matrix previously provided.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

If the residual clinical risk is unacceptable, has the developer identified additional clinical risk control measures in order to reduce the clinical risk?

ORC_ERC_OCSA_CRE7

 Further Information

Guidance/Context

If as a result of mitigation residual risk is till unacceptable then any unacceptable residual risk should be further mitigated against.

Response Type

Multiple Option

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

If the Developer has determined that no suitable risk control measures are possible, have they conducted a clinical risk benefit analysis of the clinical risk?

ORC_ERC_OCSA_CRE8

 Further Information

Guidance/Context

On occasion, a developer will run out of practical measures to implement as risk control. On those occasions, and when the clinical risk remains at an unacceptable level, the developer needs to carry out a risk benefit analysis of the app.

Response Type

Multiple Option

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the Developer’s analysis shown that the clinical benefits of the intended use outweigh the residual clinical risk?

ORC_ERC_OCSA_CRB1

 Further Information

Guidance/Context

N/A

Response Type

Multiple Option

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the Developer implemented the clinical risk control measures identified? (except where these are to be implemented by another organisation.)

ORC_ERC_OCSA_CRCM1

 Further Information

Guidance/Context

This question is looking to identify if control measures are active, where possible, this can be checked by the assessor.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Have the clinical risks from all identified hazards been considered and accepted?

ORC_ERC_OCSA_CRCM2

 Further Information

Guidance/Context

Any remaining risks that are left should be at an acceptable/low level.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Have any hazard rating reductions been fully justified?

ORC_ERC_OCSA_CRCM3

 Further Information

Guidance/Context

When hazard ratings are reduced, there should be a reason behind this. Normally, that would be due to the mitigations put in place. This question ensures that any risk rating reductions have a reason behind them.

Response Type

Yes/No

Answer Criteria

If justifications have been outlined, answer yes.
If the hazard log does not have any reduced hazard rating, answer NA.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Usability & Accessibility

Design and Development

This considers the design and development of the app and whether it follows any recognised app design standards, such as WC3, WCAG 2.0 AA, WCAG 2.1 AA, ISO 9241, Apple HIG, or Android App Quality Guidelines. The NordDEC also considers whether there was any user involvement during the development of the app, user involvement in testing, or if any features were based on user feedback.

Question

Question Reference Source

Is there a statement within the app outlining compliance with any currently recognised app design standards?     

ORC_DE01

 Further Information

Guidance/Context

This information is likely to be found in the accessibility statement, it may also be found in the about section within the app or on the developer website. Choose from the available options, or click none if none apply.

Response Type

Multiple Choice

Answer Criteria

- WC3
- WCAG 2.0 AA
- WCAG 2.1 AA
- ISO 9241
- Apple HIG
- Android App Quality Guidelines

Logic

There is no disablement logic written for this question.

Scoring Impact

High value applied if yes.

Is there a statement about user feedback during design/development?

ORC_DT01

 Further Information

Guidance/Context

This question is to determine if relevant users/user feedback have been considered, in the design of the app - BEFORE or AFTER the app was released.

Response Type

Yes/No

Answer Criteria

YES: If the developer has added features based on user feedback, and states what has changed. This can be before or after the app was published, but changes must have been made.

If the app was designed by the developer to remedy a problem they were suffering, or caring for someone suffering.

If an app is developed by doctors, for doctors.

If the app makes changes based on data collected, or users updating database e.g. MyFitnessPal.

If the app has undergone a survey/pilot study involving users, and changes were made based on the outcomes.

NO: If the app states “may add features based on feedback” - it needs to state which specific features were added.

Logic

There is no disablement logic written for this question.

Scoring Impact

Medium value applied if yes.

Is there any evidence of user involvement in testing?   

ORC_DT02

 Further Information

Guidance/Context

This question is to determine if there is any evidence that users have tested, or provided feedback on the app AFTER it was released.

Response Type

Yes/No

Answer Criteria

YES: If there is a case study on the developer website.

If there was a Beta version of the app available before the app went live.

If user feedback is shown on the website showing the app has been beneficial to users (e.g. 87% of patients have shown improvement from using the app).

Any other evidence of user testing rather than opinions from the general public.

Any evidence of indicated user benefit (if you have selected this in EE02).

No: If the only user feedback is from app store reviews.

Logic

There is no disablement logic written for this question.

Scoring Impact

Medium value applied if yes.

(Web-Apps Only)
Has the App been designed to work on Mobile Devices and Tablets?

ORC_U01a

 Further Information

Guidance/Context

This question is only applicable for web app reviews. It is identifying if the app can be used on multiple devices without formatting or other issues.

Response Type

Yes/No

Answer Criteria

YES: If it is available for both tablet and smartphone.

Logic

There is no disablement logic written for this question.

Scoring Impact

Medium value applied if yes.

Accessibility

Accessibility is important to consider, as the app should be accessible to all users regardless of their specific needs. The NordDEC considers whether the app is customisable to suit certain needs, such as poor sight or hearing impairments. If the app uses any specialist or medical terms, these should be clearly explained to the user.

Question

Question Reference Source

Can the user change the font size in-app/does the app respond to device preferences?    

ORC_U04

 Further Information

Guidance/Context

This is a key aspect for improving accessibility of apps to demographics with accessibility needs.

Response Type

Yes/No

Answer Criteria

YES: The app responds to font size changes in the device, or the font size can be changed from within the app.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low value applied if yes.

Does the app provide support for users with poor sight?     

ORC_U07

 Further Information

Guidance/Context

This question aims to address whether the app developer has considered the accessibility needs of people with perceptual impairments - specifically poor sight, e.g. blind, colour blind, poor vision.

Response Type

Yes/No

Answer Criteria

YES: If the app provides audio description, or visual descriptions of pictures.

If the app uses Voice Over (iOS) or Text to Speech (Android).

If there is the ability to change the font size, or zoom in, or make colour adjustments.

Logic

There is no disablement logic written for this question.

Scoring Impact

Medium value applied if yes.

Does the app provide support for users with hearing difficulty?        

ORC_U08

 Further Information

Guidance/Context

This question aims to address whether the app developer has considered the accessibility needs of people with perceptual impairments - specifically poor hearing, e.g. deafness, or hard of hearing.

Response Type

Yes/No

Answer Criteria

YES: If the app provides audio description and it is possible to adjust the volume of Text to Speech/voiceover (if using in-built text to speech on iOS and Android the volume can be adjusted).
If subtitles are available for video/audio/in game dialogue.

Logic

There is no disablement logic written for this question.

Scoring Impact

Medium value applied if yes.

(Web-Apps Only)
Does the App provide users with an Accessibility Statement?

ORC_U26

 Further Information

Guidance/Context

Question only applicable for web app reviews.
This would be available on the developers website, typically at the bottom of the page where links for the Privacy Policy and Terms & Conditions are kept.

Response Type

Yes/No

Answer Criteria

Yes if an accessibility statement is provided for the product, separately from any other terms and conditions.

Logic

There is no disablement logic written for this question.

Scoring Impact

Medium value applied if yes or medium risk applied if no.

Usability

This also ties in to the accessibility of the app, including further customisation options. The NordDEC identifies if the app has any functions to aid navigation, such as a home button, back button, help button or search feature. If the app utilises push or email notifications, the NordDEC identifies whether the user has options to manage these for their own preference or privacy, both at app level and at device level. Finally, if there are any bugs identified during evaluation, this will be flagged. If the app contains a forum, then we look for a statement to ensure that forum content is moderated.

Question

Question Reference Source

Can the user change the presentation theme?

ORC_U06

 Further Information

Guidance/Context

This question is looking to see if the app developer has considered accessibility needs for a breadth of audiences, i.e. not specific to certain impairments. Or if they have considered the usability/customisation of the product. If users are able to tailor an app to their own preferences/needs will increase usability and accessibility - for example, if they can change the language to their preferred language, change the units to something more understandable, or change the colour scheme to something easier on their vision.

Response Type

Yes/No

Answer Criteria

YES: If any visible changes can be made, which are not otherwise mentioned.
I.e. the ability to change colours, profile pictures, language, units, music etc.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low value.

Does the app include the following functions:

·         Home/Menu button

·         Back button

·         Help/About button

·         Search button

ORC_U32

 Further Information

Guidance/Context

By having familiar buttons such as home/help/search/about users can more easily navigate the app, as users will be familiar with things such as a magnifying glass representing the search feature.

Caveat - this question will be removed from V6.1/V7, as we feel it doesn’t add much value or information.

Response Type

Multiple Choice

Answer Criteria

Home: a button from any page back to the original page the app opens on. This should be accessible from all pages.
Back: the ‘back’ on an android phone does not count, it must be in-app.
Help: a tutorial, or how to use the app or certain features.
Search: a search bar, or any other way to filter and find information..

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Are any medical, specialist or technical terms explained clearly to the user?

ORC_U15

 Further Information

Guidance/Context

This question is to improve accessibility of the product, ensuring the app developer has considered the needs of users who may have a lower digital literacy, a lower reading level, or a lack of specialist knowledge. Explaining all key terms improves the accessibility of the product, regardless of knowledge level.

Response Type

Yes/No

Answer Criteria

YES: An instruction of how to do an exercise, even if it’s a picture.
A glossary, or any definition of specialist terms (only 1 definition is needed).

Logic

There is no disablement logic written for this question.

Scoring Impact

Medium value if answered yes.

Does the app send push notifications?

ORC_D29

 Further Information

Guidance/Context

This question looks to capture if the app has certain functions.

Response Type

Yes/No

Answer Criteria

YES: If the app sends push notifications to the device, not just in-app.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the app send email notifications?

ORC_D30

 Further Information

Guidance/Context

This question looks to capture if the app has certain functions.

Response Type

Yes/No

Answer Criteria

YES: If the app sends email notifications relating to the user’s use of the app, personalised.

NO: If the only emails are marketing/newsletters.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the user have options to manage the notification settings (push/email) within the app for convenience/privacy? 

ORC_D31

 Further Information

Guidance/Context

Allowing users the control to manage their notifications increases the likeability/usability of the product. Apps which show notifications/pop ups with sensitive information may not be preferable to a user, if they have no way to disable this. Additionally, it can be simply annoying.

Response Type

Yes/No

Answer Criteria

YES: If there is the ability to toggle notifications, or choose the time they are sent from within the app.

Logic

There is no disablement logic written for this question.

Scoring Impact

Low value applied if answered yes.

Does the app inform the user how to manage notification settings for convenience/privacy (to prevent info being shown if device is locked but on show)? (android only question)

ORC_D32

 Further Information

Guidance/Context

This is related to how a user can control notifications via the device settings. The previous question is within the app, this question is focused at the device level. This helps ensure that no information is shown on the lock screen that may be private to the user. iOS does this by default with the notifications pop-up that appears upon initial opening of an app.

Response Type

Yes/No

Answer Criteria

YES: If you can control the privacy of notifications. This is almost always yes for iOS (if it sends a pop-up), and almost always no for android.
YES: Android - If they provide instructions of how to disable notifications within the device settings

Logic

There is no disablement logic written for this question.

Scoring Impact

Low value applied if answered yes.

Was there any evidence of bugs during evaluation?

ORC_U23

 Further Information

Guidance/Context

If a bug is identified it should be assessed by another person/device to confirm. If a bug is confirmed then the Developer should be notified.

Response Type

Yes/No

Answer Criteria

YES: If the app crashes or shuts down.
If a link leads to the wrong place.
If specific buttons don’t work.

Logic

There is no disablement logic written for this question.

Scoring Impact

High risk applied if yes.

Support

Support is a key area of this section, as it is important that users are informed of ways in which they can contact the developer should they have any problems or questions with the app. The NordDEC also identifies what type of support is offered to users, and if there is a commitment from the developer to respond to any user queries. We would expect to see that the type of support offered is appropriate to the app level - a higher level app would therefore require a more sophisticated offer of user support.

Question

Question Reference Source

If there is a forum, is there a statement within the app that the forum content is moderated?

ORC_FC03

 Further Information

Guidance/Context

f there is a forum, or any peer communication between users, it is important that there is moderation, guidelines or safeguards in place, to protect users from harmful or incorrect content. This introduces an element of safety to ensure the users aren’t exposed to false information in relation to their health. Many users will feel more comfortable using forums if they know the content is moderated.

Response Type

Yes/No

Answer Criteria

YES: If there is mention of moderation, community guidelines, or if users are asked to report offensive material.
If there is mention of the developer reserving the right to modify or remove content.

NO: If there is a statement about the risks of following third party links and no mention of the above.

Logic

Logic

There is no disablement logic written for this question.

Scoring Impact

Medium risk applied if yes.

Is there a statement about how to report issues to the developer?

ORC_U24

 Further Information

Guidance/Context

Users should be able to raise any issues easily to developers. This can be identified either within the app or on the developer website ( it needs to be clear the details for contacting are about the app or the website is for the app only).

Response Type

Yes/No

Answer Criteria

YES: If a contact method is provided within the app, or accompanying website.
If there is any way for the user to contact the developer electronically from the app.

NO: If the only way to contact is to provide an app store rating.
Email addresses provided as standard on the Play Store do not qualify.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

What kind of support is offered?

ORC_U33

 Further Information

Guidance/Context

Support should be available at an equivalent quantity to the complexity of the app. This needs to be within the app or on the website, NOT an email address on the Play Store.

Response Type

Multiple Choice

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Varying risk applied depending on ESF tier of app. Higher ESF tiers are required to have better support features.

Is there any statement within the app about the developer’s commitment to addressing problems reported to them? (e.g. timescales to respond, commitment to eradicate reported bugs and faults)       

ORC_U25

 Further Information

Guidance/Context

This question is looking to ensure that app developers are providing an SLA of sorts to their users, that they are committing to responding to and/or resolving any queries in a timely manner. Users may be more likely to reach out for support, if they know when they can expect a response.

Response Type

Yes/No

Answer Criteria

YES: If a time frame to respond is specified, it can be “we will get back to you as soon as possible”.
If the website states this alongside a contact method.

NO: If the statement does not give a timeframe or indication they intend to respond e.g. “we will get back to you” is not specific enough for a timescale.

Logic

There is no disablement logic written for this question.

Scoring Impact

Medium value applied if yes.

Security & Technical Stability

Technical Stability

Question

Question Reference Source

Does the App connect to an internet-based API (e.g. App Developer Web Service, Social Media, Adverts)?

ORC_ERC_OTS_C01

 Further Information

Guidance/Context

This question is asked to help determine the needs for technical security.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

List the APIs

ORC_ERC_OTS_C02

 Further Information

Guidance/Context

The APIs identified by this question are important to assess the appropriateness of the penetration testing carried out on the app.

Response Type

Free Text

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the App connect to a medical device?

ORC_ERC_OTS_C03

 Further Information

Guidance/Context

If an app connects to a medical device enhanced consideration needs to be taken surrounding technical security. Further consideration may then be needed to surrounding multifactor authentication.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the App connect to healthcare services?

ORC_ERC_OTS_C04

 Further Information

Guidance/Context

If an app connects to healthcare services enhanced consideration needs to be taken surrounding technical security. Further consideration may then be needed to surrounding multifactor authentication.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the App operate without wi-fi?

ORC_ERC_OTS_C05

 Further Information

Guidance/Context

This question is needed to help understand the technical architecture of the app so that it can be ensured that the approach to testing is appropriate.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the App operate without cellular network?

ORC_ERC_OTS_C06

 Further Information

Guidance/Context

This question is needed to help understand the technical architecture of the app so that it can be ensured that the approach to testing is appropriate.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is the platform Web based or Mobile?

ORC_ERC_OTS_D04

 Further Information

Guidance/Context

This question is needed to help understand the technical architecture of the app so that it can be ensured that the approach to testing is appropriate and it helps determine what is the appropriate MASVS level is appropriate to the platform.

Response Type

Multiple Option

Answer Criteria

Mobile

Web

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the App access, process or store Personal and/or Sensitive Data?

ORC_ERC_OTS_D01

 Further Information

Guidance/Context

This question helps determine what is the appropriate MASVS level is appropriate to the platform.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is sensitive data persisted to the mobile device?

ORC_ERC_OTS_D02

 Further Information

Guidance/Context

This question helps determine what is the appropriate MASVS level is appropriate to the platform. This is required to check the appropriateness of the Penetration test (PEN Test)

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the App access, process or store Personal and/or Sensitive Data?

ORC_ERC_OTS_D03

 Further Information

Guidance/Context

This question helps determine what is the appropriate MASVS level is appropriate to the platform. This is required to check the appropriateness of the Penetration test.

Response Type

Multiple Option

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

What Permissions does the App request?

ORC_ERC_OTS_P01

 Further Information

Guidance/Context

This is only relevant if the app is a mobile device. It is needed to help inform the appropriateness of the penetration test. Some permissions will be inappropriate given the functionality of the app e.g. camera. Best practice is for access to be explicitly turned off.

Response Type

Free Text

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the App provide Alerts or Notifications?

ORC_ERC_OTS_OTF01

 Further Information

Guidance/Context

This question allows assessors to understand the risks of the app and therefore take a proportional approach to the evidence needed to passTechnical Security.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the App provide Suggestions?

ORC_ERC_OTS_OTF02

 Further Information

Guidance/Context

This question allows assessors to understand the risks of the app and therefore take a proportional approach to the evidence needed to pass Technical Security.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the App undertake calculations?

ORC_ERC_OTS_OTF03

 Further Information

Guidance/Context

This question allows assessors to understand the risks of the app and therefore take a proportional approach to the evidence needed to pass Technical Security.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Are the source code and any configuration items for the product version controlled with all changes audited?

ORC_ERC_OTS_PSL01

 Further Information

Guidance/Context

Version control of source-code and configuration is a fundamental requirement of robust software development and applications. Evidence should include documented processes and procedures if they exist, and screenshots of any tools that are used.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Medium value applied if yes.

Provide details of any associated processes / procedures and tools that are used.

ORC_ERC_OTS_PSL02

 Further Information

Guidance/Context

This question is to allow a description of the previous question to be provided.

Response Type

Free Text

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Do you have the capacity to rollback to previous versions of your product?

ORC_ERC_OTS_PSL03

 Further Information

Guidance/Context

This will be achievable if appropriate version control of source-code and configuration is in place. Evidence should include documented processes and procedures if they exist, and screenshots of any tools that are used.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Low value applied if yes.

Provide details of any associated processes / procedures and tools that are used.

ORC_ERC_OTS_PSL04

 Further Information

Guidance/Context

This question is to allow a description of the previous question to be provided.

Response Type

Free Text

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Are the processes for accepting and responding to technical faults from end users appropriate?

ORC_ERC_OTS_PSL05

 Further Information

Guidance/Context

Having a mechanism to capture and respond to user feedback is essential to be able to provide a responsive service to users

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Low value applied if yes.

Do you provide online support for user queries?

ORC_ERC_OTS_PSL06

 Further Information

Guidance/Context

Having a mechanism to capture and respond to user feedback is essential to be able to provide a responsive service to users

Response Type

Free Text

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Do you proactively monitor running of systems and system components to automatically identify faults and technical issues?

ORC_ERC_OTS_PSL07

 Further Information

Guidance/Context

Having a mechanism to capture and respond to user feedback is essential to be able to provide a responsive service to users.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Low value applied if yes.

Provide details of any associated processes / procedures and tools that are used.

ORC_ERC_OTS_PSL08

 Further Information

Guidance/Context

Evidence should include documented processes and procedures if they exist, and screenshots of any tools that are used.

Response Type

Free Text

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Do you have a documented roadmap for future development of your product?

ORC_ERC_OTS_PSL09

 Further Information

Guidance/Context

Having a live roadmap demonstrates that the app will be improved in response to emerging requirements, there is provision to manage technical debt and changes required to respond to new security threats and vulnerabilities will be addressed.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Low value applied if yes.

Provide details of planned development, technical updates.

ORC_ERC_OTS_PSL10

 Further Information

Guidance/Context

Having a live release cycle demonstrates that the app will be improved in response to emerging requirements, there is provision to manage technical debt and changes required to respond to new security threats and vulnerabilities will be addressed.

Response Type

Free Text

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the Developer provide details of how they will ensure the continued availability of their product?

ORC_ERC_OTS_PSL11

 Further Information

Guidance/Context

This is necessary as as User technology updates the product needs to remain viable for the people that use it.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Low value applied if yes.

Do you have a plan for decommissioning your product?

ORC_ERC_OTS_PSL12

 Further Information

Guidance/Context

This is important to ensure that user data is dealt with appropriately. It demonstrates that the app provider had a full appreciation of a product lifecycle.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Low value applied if yes.

Describe your processes for decommissioning your product and dealing with any identifiable data.

ORC_ERC_OTS_PSL13

 Further Information

Guidance/Context

Evidence should include documented processes and procedures if they exist, and screenshots of any tools that are used.

Response Type

Free Text

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Do you have a plan for dealing with any identifiable data in the event that an individual stops using your product? For example by installing or unsubscribing.

ORC_ERC_OTS_PSL14

 Further Information

Guidance/Context

Evidence should include documented processes and procedures if they exist, and screenshots of any tools that are used to purge data.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Low value applied if yes.

Provide details of any associated processes / procedures and tools that are used.

ORC_ERC_OTS_PSL15

 Further Information

Guidance/Context

This question is to allow a description of the previous question to be provided.

Response Type

Free Text

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the organisation follow any formal testing standards

ORC_ERC_OTS_PSL16

 Further Information

Guidance/Context

Evidence should include documented processes and procedures if they exist, and screenshots of any tools that are used. Evidence of formal certification can also be provided.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Low value applied if yes.

Provide details of any associated processes / procedures and tools that are used.

ORC_ERC_OTS_PSL17

 Further Information

Guidance/Context

This question is to allow a description of the previous question to be provided.

Response Type

Free Text

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

For each of the following if they are carried out please describe the people / roles that are involved, the processes that they work to even if they are informal.

ORC_ERC_OTS_PSL18

 Further Information

Guidance/Context

Formal test plans, checklists and screenshots of tools can be provided as evidence.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Unit

ORC_ERC_OTS_PSL19

Regression

ORC_ERC_OTS_PSL20

 End-to-end / Integration

ORC_ERC_OTS_PSL21

User Acceptance

ORC_ERC_OTS_PSL22

A/B

ORC_ERC_OTS_PSL23

PEN / Vulnerability

ORC_ERC_OTS_PSL24

Testing across devices

ORC_ERC_OTS_PSL25

Load / Performance

ORC_ERC_OTS_PSL26

Security

ORC_ERC_OTS_PSL27

Other non-functional tests

ORC_ERC_OTS_PSL28

Other testing

ORC_ERC_OTS_PSL29

Has the Developer provided sufficient evidence to satisfy all the requirements of the product's testing?

ORC_ERC_OTS_PSL30

 

Technical Security

Question

Question Reference Source

Does the organisation have ISO27001:2013 accreditation?

ORC_ERC_SEC_ORG1

 Further Information

Guidance/Context

ISO/IEC 27001 is an international standard on how to manage information security. A developer demonstrating that they are accredited with ISO27001 demonstrates that they safeguards their and user data appropriately and in line with best practice.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is the certification body in the UKAS list of ISO27001:2013 certification bodies?

ORC_ERC_SEC_ORG2

 Further Information

Guidance/Context

UKAS is the National Accreditation Body for the United Kingdom.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Medium value applied if yes.

Please provide Statement of Applicability?

ORC_ERC_SEC_ORG3

 Further Information

Guidance/Context

The Statement of Applicability is the foundational document for ISO 27001. It defines which of the suggested 114 controls that will be implemented and how, as well as if needed the reasons why certain controls have not been implemented.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the scope include product and associated services?

ORC_ERC_SEC_ORG4

 Further Information

Guidance/Context

Within the ISO 27001 documentation is should be made clear that it covers

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is the Application a Native Application for a Mobile Device?

ORC_ERC_SEC01

 Further Information

Guidance/Context

This question informs the OWASP guidelines that should be adopted in the PEN test. e.g. ASVS vs MASVS.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is the Application a Web Application?

ORC_ERC_SEC02

 Further Information

Guidance/Context

This question informs the OWASP guidelines that should be adopted in the PEN test. e.g. ASVS vs MASVS.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Are Web API’s accessed?

ORC_ERC_SEC03

 Further Information

Guidance/Context

This question informs the scope of the PEN test e.g. web-API end points should be in-scope if the application is deemed high-risk.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the App access, process or store Personal and/or Sensitive Data?

ORC_ERC_SEC04

 Further Information

Guidance/Context

This question informs the OWASP Level required for the PEN test.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is sensitive data persisted to the mobile device?

ORC_ERC_SEC05

 Further Information

Guidance/Context

This informs the OWASP Level required for the PEN test

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

What Permissions does the Application request?

ORC_ERC_SEC06

 Further Information

Guidance/Context

Some permissions will be inappropriate given the functionality of the functionality of the application e.g. camera. Best practice is for access to explicitly turned off and this isn’t always the case depending upon the approach to software development.

Response Type

Multiple Choice

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

OWASP Level is required in order to review evidence.

IF Mobile = Y

IF 'Personal and /or Sensitive Data is accessed, processed or stored’ = Y

OWASP Level then MASVS = 2

IF Sensitive data is persisted to the device then MASVS = 2+R

ELSE

OWASP Level then MASVS = 1

IF Web = Y

IF 'Personal and /or Sensitive Data is accessed, processed or stored' = Y

OWASP Level then ASVS= 2

ELSE

OWASP Level then ASVS= 1

ORC_ERC_SEC07
(What OWASP Level is the App)

Does the Application connect to a Medical device?

ORC_ERC_SEC08

 Further Information

Guidance/Context

This questions helps inform the assessor of the risks of the app and therefore take a proportional approach to the evidence needed to pass Technical Security.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the Application connect to a healthcare service?

ORC_ERC_SEC09

 Further Information

Guidance/Context

This questions helps inform the assessor of the risks of the app and therefore take a proportional approach to the evidence needed to pass Technical Security. We would also expect evidence of formal integration testing.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the Application provide Alerts or Notifications?

ORC_ERC_SEC10

 Further Information

Guidance/Context

This questions helps inform the assessor of the risks of the app and therefore take a proportional approach to the evidence needed to pass Technical Security.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the Application provide Suggestions?

ORC_ERC_SEC11

 Further Information

Guidance/Context

This questions helps inform the assessor of the risks of the app and therefore take a proportional approach to the evidence needed to pass Technical Security.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the Application undertake Calculations?

ORC_ERC_SEC12

 Further Information

Guidance/Context

This questions helps inform the assessor of the risks of the app and therefore take a proportional approach to the evidence needed to pass Technical Security.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Does the Application support in-App purchases?

ORC_ERC_SEC13

 Further Information

Guidance/Context

This questions helps inform the assessor of the risks of the app and therefore take a proportional approach to the evidence needed to pass Technical Security.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has a Security Assessment been undertaken by an accredited external third-party?

ORC_ERC_SEC14

 Further Information

Guidance/Context

This provides assurance that the PEN test will have been scoped appropriately and the methodology will also be appropriate.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Is the external third-party a CREST / APMG / CHECK registered supplier?

ORC_ERC_SEC15

 Further Information

Guidance/Context

If yes, we can likely trust the outcome of the PEN test report. CREST and CHECK are certified bodies.

Response Type

Multiple Option

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

High risk applied if answered No.

Does the scope of the report cover the full Technical Architecture of Application?

ORC_ERC_SEC16

 Further Information

Guidance/Context

Take a look at the APIs which the PEN test assesses, and if this covers the full scope of the product. e.g. you might expect to see APIs related specifically to the mobile app, rather than an unrelated part of the website.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has an industry-standard been used for the risk model in the associated PEN/Vulnerability testing?

ORC_ERC_SEC17

 Further Information

Guidance/Context

This provides assurance that the PEN test has been executed professionally e.g. Common Vulnerability Scoring System (CVSS)

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Have all ‘Medium’ Risks / Issues identified been mitigated and resolved; and can this be demonstrated through retesting within six weeks from the original PEN / Vulnerability testing?

ORC_ERC_SEC18

 Further Information

Guidance/Context

Evidence should include the full version of the original PEN test report and any retest.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Has the Code-Level Security Assessment been undertaken against the correct OWASP Level?

ORC_ERC_SEC19

 Further Information

Guidance/Context

This level will be detailed in the Security Assessment report and any associated PEN testing report. This level should match the deemed appropriate level by the earlier questions.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

Low value applied if answered Yes.

Is the methodology for the Security Review proportional to the attack service and risk of the Application?

ORC_ERC_SEC20

 Further Information

Guidance/Context

The scope and methodology should be proportional to the associated risk.

Response Type

Yes/No

Answer Criteria

N/A

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

 

The project is run by N!P, jointly funded by Nordic Innovation and Nordic healthtech industry and powered by ORCHA.