Digital Health Assessment Framework V1.0

Created by

Last updated: 13 September 2022, 10:08

With more than 86 million Americans already using a health or fitness app, digital health brings new possibilities for the healthcare industry.

Yet, in a field of 365,000 products, where the vast majority fall outside of existing regulations, such as the medical device regulations, federal laws and government guidance, there has been no clear way to determine if a product is safe to use. This is stopping the national adoption of digital health, particularly in the fields of condition management, clinical risk assessment and decision support.

The Digital Health Assessment Framework has been created to:

Be an open, objective framework, accessible for anyone to use.
Support the adoption of high-quality digital health technologies.
Help healthcare professionals and consumers make better-informed decisions.
Meet the specific needs and requirements of the US market.

The Framework includes components, specific to the needs of the US, crafted across across four fields:

Data and Privacy
Clinical Assurance and Safety
Usability and Accessibility
Technical Security and Stability

Rather than try to reinvent the wheel, the Framework recognizes and points to relevant existing US regulations, and applies several leading international standards and frameworks, ISO 82304-2 in Europe, Digital Technology Assessment Criteria (DTAC) and NICE evidence standards framework in the UK, and DiGA in Germany.

The American College of Physicians (ACP) and the American Telemedicine Association (ATA) collaborated with industry leaders to develop the framework. This includes the Organization for the Review of Care and Health Apps (ORCHA), which has over six years of experience and is equipped to assess digital health products at scale. ORCHA has evolved frameworks for the deployment of digital health in the UK, across Europe and the Middle East.

The Digital Health Assessment Framework is intended to be an open framework, accessible for anyone to use, to support the adoption of high-quality digital health technologies. The Framework will be updated regularly with input from healthcare providers, consumers, technology developers and other stakeholders to reflect changes in clinical practice, and the latest guidelines and best practices.

The framework is managed and published by an independent steering committee, featuring members from across the industry. If you would like to join the committee, or find out more about the framework, please do get in touch, we’d love to hear from you.

For more information visit: dhealthframework.org

Development Collaborators

INRODUCTION
- Assessment Domains
- Value and Risk Points
SCENE SETTERS
DATA & PRIVACY
CLINICAL ASSURANCE & SAFETY
USABILITY AND ACCESSIBILITY
TECHNICAL SECURITY & STABILITY (ENHANCED REVIEW COMPONENT)
- Technical Security

INRODUCTION

The Digital Health Assessment Framework ( ‘DHAF’) is designed to assess Digital Health Technologies in the form of native apps, web apps or websites. This can include ‘wellness’ oriented apps as long as there is a clear health related focus and benefit.

The DHAF is applied independently of the Developer and means the product itself is assessed (app, web app or website), any supporting website (for native apps) and the relevant ‘app store’ entry (for native apps).

Information is also derived at the time of review via general ‘google’ searches of the relevant product to check, for example, references to Clinical Trials or Studies. These are the assessment sources (“Assessment Sources”) and only information that is available publicly through these sources is considered in the assessment.

Sometimes the evidence being sought for assessment does exist but has not been made available to end users. We do not believe that this is appropriate for the types of information assessed and will only take into account information that can be freely accessed by end users.

A fundamental principle of the DHAF is that all the assessment components within it are constructed to enable an objective and evidence-based analysis of the app. This means that the process between an assessment and quality assurance signoff should provide an objective consensus.

Assessment Domains

The DHAF examines an app’s compliance with relevant regulation, guidelines and best practice in three distinct areas (‘Domains’). These are – Data & Privacy, Clinical Evidence & Safety and Usability & Accessibility.

In addition the DHAF contains an ‘Enhanced Component’ to assess Technical Security & Stability - this domain area requires additional interaction with a DHT Developer to assess the relevant elements. It is envisaged that individual Digital Health Libraries will adopt this and other bespoke Enhanced Review Components dependent on local basis dependent on application.

Each of these Domains has been developed by relevant subject matter experts and consist of a series of objective questions and strict guidance on the criteria that justifies either option. To achieve this relevant regulation and guidance is ‘deconstructed’ into smaller pieces that can be objectively assessed and then built up into a broader picture of compliance. We have done this for example in relation to the intricacies of HIPAA compliance. This approach also allows us to avoid duplication in the assessment process as it is often the case as there are many overlapping principles and data elements.

The DHAF draws on existing and emerging Digital Health assessment frameworks both in the US and Internationally. Where we haven’t used any particular aspect of an alternative framework it will be because either:

· The assessment component is not capable of objective assessment; or

· The assessment component would require evidence from a Developer that isn’t habitually made publicly available (i.e. Risk Logs, Security Assessments etc); or

· The nature of the assessment component is too onerous for the ‘Foundation’ nature of the DHAF and is better considered as an Enhanced Component.

Value and Risk Points

The scoring is made up of Value earning points and Risk earning points.
Each scoring question has either a Risk implication or a Value implication
The quantum of the Risk or Value implication is decided by the relevant tariff which range from small, medium, high or exceptionally high in the Risk area and small, medium or high in the Value area.
The following table sets out the actual numeric value of each Tariff:

Tariff	Risk	Value
Small	10	5
Medium	20	10
High	40	20
Exceptionally High	80	-

In addition to the base tariff, some risk and value related questions attract a multiplier that will increase the relevant tariff based on certain related app characteristics.
Maximum risk can be applied based on responses for certain questions. Maximum risk is applied to a whole section (i.e Data), rather than an individual question. It is the sum of all the risk points that could be applied if were not for the questions being disabled by earlier responses.

SCENE SETTERS

The USDHA begins with a series of questions to capture an app's core purpose and functionality. These include the target audience, the type of data the app collects and the app’s primary functions and features. None of the scene setter questions is intended to have any scoring or risk implications and are purely to decide on the line of enquiry further in the review.

The scene setters are grouped into distinct areas, all of which can be seen below.

Note all references to the app/App are references to the relevant Digital Health Product. These can be mobile applications or web-based applications.

Every question within scene setters does not have a scoring value.

App Characteristics

Question Set

Is the App health focused?

ORC_SS01

Further Information

Guidance/Context

The purpose of this question is to identify apps which are within the DHAF scope of assessment. This includes any apps which have a clear health or medical purpose, are condition specific, or have a valid place in a clinical setting. This also includes wellness apps if they have a clear focus on a particular need or condition, eg. yoga apps for pregnancy. Apps which have no clear or specific health focus are excluded, eg. generic meditation apps.

Response

Yes / No

Answer Criteria

Yes: If the app has a specific health, fitness, lifestyle purpose or claim. If the app is condition specific. If the app has a clear place in a clinical setting.

No: If app has no obvious health purpose (e.g. voice recorder, screen recorder, keyboard, a timer app, recipe books)If the app has no health purpose and does not relate to any kind of health condition (e.g. general meditation apps). Fitness app where exercises are not designed to prevent a specific condition e.g. a circuit training app with no health claims.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Data - Data Types, Data Collection and Data Sharing

Question Set

Further Information Guidance/Context The purpose of this question is to identify if the app collects data so the relevant data questions are disabled appropriately. Response Yes/No Answer Criteria Yes: If any data is collected by or through the app, in any way. Including data such as usage data, cookies etc. No: If no data is collected from the user or the app. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Does the App collect data?	ORC_D01
What type of data is collected by the App?	ORC_DT10
Further Information Guidance/Context This question aims to identify what type of data the app collects. This is answered based on what information can be submitted into the app and also what is visible in the privacy policy. To select data items from the privacy policy it must be clear that the data items are are collected via the app and not via an associated website. For instance if the privacy policy states “when you use our services…”, it is essential that the term ‘services’ is checked to see if it is referring to the app or not. If ‘services’ does refer to the app then the data items mentioned following that statement should be selected but if ‘services’ simply refers to an associated website then the data items should not be selected. If cookies are mentioned in the privacy policy, they are only included if they relate to the app. Biometric data is only selected if such data is directly processed by the app and/or the device’s inbuilt software isn’t used. Response Multiple Choice Answer Criteria Name (Full Name, Nickname or First Name Only), Username, Mobile Number/Device Number/Home Phone Number, Email, Full Address/Postcode, Age/DOB, Location Data, IP Address, Device IMEI Number, Other Unique Device Identifiers, General Wellness Data, General Identifier e.g. NHS No, Physical Description, Martial Status/Family/Lifestyle/Social Circumstance, Income/Financial/Tax Situation, Education/Qualifications/Professional Training/Awards, Employment/Career History, Cookies/Web Beacons etc. (used for tracking an individual’s online browsing behaviors/movements), Usage Data, Card/Payment/Financial Information, Race/Ethnic Origin, Political/Religious/Philosophical Beliefs or Opinions, Trade Union Membership, Genetic or Biometric Data (where this is used for identification purposes e.g. fingerprint or facial recognition), Physical and/or Mental Health Data, Sexual Orientation or Sex Life, Gender (self-declared or observed), Offences Committed/Alleged to have Committed/Criminal Proceedings/Outcomes/Sentence, Other Online Identifiers, Other (please specify). Logic DISABLEMENT LOGIC - Disabled if D01 is no. Scoring Impact There is no scoring impact associated with this question.
What Permissions does the app request?	ORC_ERC_OTS_P01
Further Information Guidance/Context This question is only relevant if the platform is a mobile app. You can find the answer to this question through the device’s settings and through the google play store and app store. This question helps inform the Technical Security questions which are asked later in the assessment. Response Free text Answer criteria N/A Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Are users required/able to sign up/register to use the service?	ORC_DT14
Further Information Guidance/Context If a user is required or able to sign up to use the app, it indicates that personal information is undoubtedly collected and processed as part of the service. The assessor may then have to determine whether or not the collection and processing of personal information is strictly necessary for the provision of services. Response Yes / No Answer Criteria Yes: If any part of the service provided requires a user to set up an account. Yes: If account creation is not mandatory, but is optional for the purpose of backing up information. No: If account creation is not possible in any circumstance. Logic DISABLEMENT LOGIC - Disabled if D01 is no. Scoring Impact There is no scoring impact associated with this question.
Is data collected through cookies?	ORC_DT11
Further Information Guidance/Context This question is pre-filled if 'cookies/web beacons etc.' has been selected in DT10. This will require validation from the assessor. To answer yes, any cookies mentioned in the privacy policy/cookie policy must be in reference to the relevant app and not the associated website. This question determines whether the following cookie questions will be asked throughout the assessment. Response Yes / No Answer Criteria Yes: If the privacy policy/cookie policy states the application uses cookies. Yes: If the application stops functioning when cookies are disabled through the device settings. No: If there are no mention of cookies in the app or on the privacy policy. No: If cookies are only mentioned in relation to the associated website. Logic DISABLEMENT LOGIC - Disabled if D01 is no OR if DT10 does not contain cookies/web beacons etc. Scoring Impact There is no scoring impact associated with this question.
What type of cookies are used?	ORC_DT12
Further Information Guidance/Context The type of cookies hold different level of importance on what rights must be upheld for the user and can also act as indicators as to the nature of the data collected through cookies and whether “profiling” might be occurring, making a user identifiable. Third-party (aka tracking) cookies are used to collect data based on online behavior. This data is passed on/sold to third-party advertisers so that the information can be used for targeted adverts. User’s must be given the option to block these. Session cookies track movement around a website and can be ‘strictly necessary’. These expire as soon as you leave/close the website. An example of why these might be strictly necessary would be for online shopping, if session cookies aren’t used, then when adding to a cart and ultimately going to check-out, your cart would appear empty. Persistent/permanent cookies are used for remembering and implementing user preferences for when they return to a website. These cookies are stored on your hard disk for extended periods of time and will have varying expiration dates. Once deleted, everything customized for preferences will be forgotten. Persistent cookies are often used for computers to remember and store your login information, language selections, menu preferences, etc. If it is unclear, even through a cookie policy, as to the types of cookie being used, this might be used to apply immediate negative scoring on apps being asked questions regarding cookies in the data section. Response Multiple Choice Answer Criteria Third party: If the privacy policy or cookie policy states that third party cookies are used. Session: If the privacy policy or cookie policy states that third party cookies are used. Persistent: If the privacy policy or cookie policy states that third party cookies are used. Unclear: If the privacy policy or cookie policy is unclear to the user what types of cookies are being used. Answer Criteria Logic DISABLEMENT LOGIC - Disabled if D01 is no OR if DT10 does not contain cookies/web beacons etc OR DT11 equals no. Scoring Impact There is no scoring impact associated with this question.
Is the data (cookie and/or none cookie) collected:	ORC_DT13
Further Information Guidance/Context This question aims to determine what level the data is classified as i.e. sensitive, personal or nonpersonal. The level of data impacts what level of rights should be upheld for the user. The assessor will select the appropriate level of data by referring to the data they selected in DT10. ‘Personal’ data relates to data which can be used to identify someone whereas ‘Personal (combined)’ data refers to a number of pieces of data which when combined can be used to identify someone. Response Multiple Choice Answer Criteria Sensitive: Physical / Mental Health or Condition (past, current or future status), Sexual Life / Orientation, Political, Religious or other beliefs or opinions, Offences Committed / Alleged to have Committed / Criminal Proceedings / Outcomes / Sentence, Financial data (that might be used for payment fraud), Trade Union membership, Racial / Ethnic Origin, Genetic or Biometric Data (e.g. fingerprints / facial Recognition) for the purpose of uniquely identifying a person Personal (combined): Cookies, web beacons, flash cookies, server logs etc which track individual’s browsing behavior, Other Unique Device Identifiers eg. Device MAC Address, Name, Age/DOB, Gender (self declared or observed), Marital Status \|Family / Lifestyle / Social Circumstance, Education / Qualifications / Professional Training / Awards, Other online identifiers / Event Logs, Location Data (Travel / GPS / GSM Data / radio frequency identification tags (RFID)) Personal: Address/Postcode (full), Email Address, Mobile Phone Number / Device Number / Home Phone Number, Physical Description, Username, IP Address, General Identifier e.g. Social Security Number, Income / Financial / Tax Situation, Employment / Career History, Device IMEI No Non-personal: General Wellness data Logic DISABLEMENT LOGIC - Disabled if D01 is no. Scoring Impact There is no scoring impact associated with this question.
How is non-cookie data collected?	ORC_DC01
Further Information Guidance/Context This question aims to uncover how data is collected from an individual, this information is key to other parts of the assessment. It is particularly important to ensure that organizations make it clear to their users when there is any “blind” processing occurring. Response Multiple Choice Answer Criteria Device measurement capability: Auto GPS, motion, microphone, camera Other apps: Google fit, Apple health, Facebook Devices: Wearables, Medical devices Third party sources: Google analytics, card payment processors (stripe, PayPal) Automatically generated by the app: Usage data From Device storage: Photos saved on device From Device Information: IP address Other (please specify: Assessor to specify what other is in the comment box Logic DISABLEMENT LOGIC - Disabled if D01 is no. Scoring Impact There is no scoring impact associated with this question.
What other apps is the App connected to?	ORC_DC02
Further Information Guidance/Context Integration of information between different apps/platforms can be of value to certain individuals. That said, it can also present additional security risks. The organization should consider and mitigate these security risks when they have enabled personal information to be shared to and from their product. Response Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if D01 is no OR if DC01 does not contain ‘other apps’. Scoring Impact There is no scoring impact associated with this question.
What device(s) does the App connect to?	ORC_DC03
Further Information Guidance/Context Integration of information from devices can be of value to certain individuals. That said, it can also present additional security risks. The organization should consider and mitigate these security risks when they have enabled personal information to be shared to and from their product. Response Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if D01 is no OR if DC01 does not contain ‘other devices’. Scoring Impact There is no scoring impact associated with this question.
Can the user prevent cookie data being collected and still use the App?	ORC_DS01
Further Information Guidance/Context Under the Privacy directive any organization must minimize access restrictions, even if a user refuses to accept certain cookies. Response Yes / No Answer Criteria Yes: If the assessor has been able to leave the app > go to device browser > identify that cookies are in use > turn off all cooking relating to the service > launch and access the app again > check back on browser to ensure the previously removed/blocked cookies have not become active again. Yes: If the user is informed that they can prevent cookies and that this will only possibly prevent some functionality/access to features. No: If cookies have become active again on the device’s browser after following the steps to turn them off. No: If strictly necessary cookies are in use. No: If users are not given the option or informed how to control/prevent/turn off cookies. Logic DISABLEMENT LOGIC - Disabled if D01 is no OR if DC01 does not contain 'cookies/web beacons etc’. Scoring Impact There is no scoring impact associated with this question.
Does the disabling of cookies impact the use of the App in any way?	ORC_DS02
Further Information Guidance/Context This helps to identify whether cookies are necessary for the app to function. Response Yes / No Answer Criteria Yes: If the assessor has been able to leave the app > go to device browser > identify that cookies are in use > turn off all cooking relating to the service > launch and access the app again > has been unable to access certain features. Yes: If the user is informed that they can prevent cookies and that this will only possibly prevent some functionality/access to features. No: If disabling the cookies through the browser has no impact on functionality/access to features. No: If strictly necessary cookies are in use. Logic DISABLEMENT LOGIC - Disabled if D01 is no OR if DC01 does not contain 'cookies/web beacons etc’. Scoring Impact There is no scoring impact associated with this question.
Can/is data shared? (excluding cookies)	ORC_DS03
Further Information Guidance/Context This question determines whether the following data sharing questions are asked. Response Yes / No Answer Criteria Yes: If any data type that has been identified as collected is shared/exported from the App, on the device, in any way. This includes data being transferred and stored by the developer on external servers and includes the ability for the user to manually move data out of the app. Yes: If you have to create an account to access the app. No: If there is no data transferred from the app, to another location, on or off the device, either automatically or by manual export by the user. Logic DISABLEMENT LOGIC - Disabled if D01 is no. Scoring Impact There is no scoring impact associated with this question.
Can data be shared through a direct, manual action by the user? {e.g. by sending data via email or manually choosing to post/share something within the app etc}	ORC_DS04
Further Information Guidance/Context This question helps inform the user if they are able to share their own health data via the app. Response Yes / No Answer Criteria Yes: If any data only leaves the app or the device when the user carries out a direct action for this to occur. This action needs to be carried out every time the user wishes to share this data. (sharing data via email or sending reports manually within the app). Yes: Manually choosing to post/share something within the app. No: If data is shared without a direct action from the user. No: If data is automatically transferred following a single action of turning on a permission in the app. Logic DISABLEMENT LOGIC - Disabled if D01 is no OR if DS03 is no. Scoring Impact There is no scoring impact associated with this question.
How is the user able to manually share their data?	ORC_DS05
Further Information Guidance/Context This question helps inform the user how exact they are able to share their health data. Response Multiple Choice Answer Criteria Exporting a report to the device: If the user can store data, outside of the app, on the device itself. Exporting data to a preset email: If the user has the option to email reports/data and is taken to the device email app to send information. Exporting data to messaging services: If the user has the option to share information/data and the share options include any messaging services/apps on the device. Emails do not count as messaging services. Transfer through Bluetooth: If the user can manually share information using the devices Bluetooth. This does not include automatic transfer of data to a device that automatically connects. Nor does it include apps that continuously run Bluetooth in the background to communicate and recognize other devices (such as track and trace). Transfer through NFC (near-field communication): If the user can transfer data to another device using NFC capabilities. Manually choose to post/share something in the app: If there is any form of in app communication between two or more users, where the information/content posted is done so through the user opting to do so each time. Other (please specify): If the app is transferring through manual user intervention but the option has not been listed. Logic DISABLEMENT LOGIC - Disabled if D01 is no OR if DS04 is no. Scoring Impact There is no scoring impact associated with this question.
Is data ONLY shareable through a direct, manual action by the user? (excluding cookies)	ORC_DS06
Further Information Guidance/Context If data on the app is only manually shared, the app with external third parties of the user’s choice and is often something that is not managed or decided by the developer. Therefore if this is the only form of data sharing, and the developer cannot access or process the data away from the device that the app is on, they will be less likely to be subject to GDPR principals. Response Yes / No Answer Criteria Yes: If the only data transfers that occur are through the direct user interactions identified. No: If there is any data transferred by a means that has not been done through a direct user intervention. For example if you have selected “usage data” and this is collected automatically, then you would answer No. Logic DISABLEMENT LOGIC - Disabled if D01 is no OR if DS04 is no. Scoring Impact There is no scoring impact associated with this question.
Can the user control any automatic data sharing, through setting individual sharing preferences in the app? (excluding cookies)	ORC_DS07
Further Information Guidance/Context This question aims to find out whether users can set up automatic data sharing. Signing up through Facebook does not count, the user has to have control for example, toggle something in the app to choose ‘yes you can collect usage data’. Response Yes / No Answer Criteria Yes: If the user has control over when data is automatically shared, for example, through having individual options that can be toggled on and off, in the app. Yes: If the user can create social circles, or choose who can view/access their profile/data. Eg. Making an account private or public, or specifically selecting which members of your clinical support network can view which data. No: If the user has no choice in whether or not to sign up to the app in order to use it, i.e. if the user MUST sign up to use, or CANNOT sign up at all. Logic DISABLEMENT LOGIC - Disabled if D01 is no OR if DS03 is no OR if DS06 is yes. Scoring Impact There is no scoring impact associated with this question.
Where/With who can the user share data automatically by manually setting sharing preferences in the app?	ORC_DS08
Further Information Guidance/Context If the user can control automatic data sharing via a toggle, this question identifies with whom the user can share the data. Response Multiple Choice Answer Criteria Developer Clinician/HCP Other users Third parties: Google Fit, Apple Health, Facebook, google analytics, etc. Other devices: Wearables, scales, medical devices. Logic DISABLEMENT LOGIC - Disabled if D01 is no OR if DS07 is no. Scoring Impact There is no scoring impact associated with this question.
Is any data (excluding cookie data) shared automatically as soon as the App is accessed – based only on agreement to relevant Terms of Use or Privacy Policy?	ORC_DS09_US
Further Information Guidance/Context This question aims to identify if any automatic sharing of data occurs without any input from the user beyond agreeing to the Privacy Policy and/or T&C’s. Response Yes / No Answer Criteria Yes: The privacy policy states that data is automatically shared. No: The privacy policy / app clearly states that no data is shared without the users input. Logic DISABLEMENT LOGIC - Disabled if D01 is no OR if DS03 is no or IF DS06 is yes. Scoring Impact There is no scoring impact associated with this question.
Where/With who is data automatically shared - based only on user agreement to the developer’s Privacy Policy and/or Terms of Use?	ORC_DS10_US
Further Information Guidance/Context This question highlights with who the user’s data is automatically shared with. Response Multiple Choice Answer Criteria Developer, Clinician/HCP Other users Third parties: Google Fit, Apple Health, Facebook, google analytics, etc. Other devices: Wearables, scales, medical devices. Logic DISABLEMENT LOGIC - Disabled if D01 is no OR if DS09 is no. Scoring Impact There is no scoring impact associated with this question.
What data is automatically shared with the developer?	ORC_DS12
Further Information Guidance/Context If the policy has clearly stated that data is automatically shared with the developer but is not clear exactly what data is shared then assessors can assume that all collected data is automatically shared. Do not select Biometric data if it is using the phone's native scanner/face ID rather than an in-app one scanner. Response Multiple Choice Answer Criteria Name (Full Name, Nickname or First Name Only), Username, Mobile Number/Device Number/Home Phone Number, Email, Full Address/Postcode, Age/DOB, Location Data, IP Address, Device IMEI Number, Other Unique Device Identifiers, General Wellness Data, General Identifier e.g. NHS No, Physical Description, Martial Status/Family/Lifestyle/Social Circumstance, Income/Financial/Tax Situation, Education/Qualifications/Professional Training/Awards, Employment/Career History, Cookies/Web Beacons etc. (used for tracking an individual’s online browsing behaviors/movements), Usage Data, Card/Payment/Financial Information, Race/Ethnic Origin, Political/Religious/Philosophical Beliefs or Opinions, Trade Union Membership, Genetic or Biometric Data (where this is used for identification purposes e.g. fingerprint or facial recognition), Physical and/or Mental Health Data, Sexual Orientation or Sex Life, Gender (self-declared or observed), Offences Committed/Alleged to have Committed/Criminal Proceedings/Outcomes/Sentence, Other Online Identifiers, Other (please specify). Logic DISABLEMENT LOGIC - Disabled if D01 is no OR if DS06 is yes. OR if DS08 does not contain developer AND DS10 does not contain developer. Scoring Impact There is no scoring impact associated with this question.
What data is automatically shared with physicians / healthcare professionals?	ORC_DS13_US
Further Information Guidance/Context If the policy has clearly stated that data is automatically shared with the physicians / healthcare professionals but is not clear exactly what data is shared then assessors can assume that all collected data is automatically shared. Do not select Biometric data if it is using the phone's native scanner/face ID rather than an in-app one scanner. Response Multiple Choice Answer Criteria Name (Full Name, Nickname or First Name Only), Username, Mobile Number/Device Number/Home Phone Number, Email, Full Address/Postcode, Age/DOB, Location Data, IP Address, Device IMEI Number, Other Unique Device Identifiers, General Wellness Data, General Identifier e.g. NHS No, Physical Description, Martial Status/Family/Lifestyle/Social Circumstance, Income/Financial/Tax Situation, Education/Qualifications/Professional Training/Awards, Employment/Career History, Cookies/Web Beacons etc. (used for tracking an individual’s online browsing behaviors/movements), Usage Data, Card/Payment/Financial Information, Race/Ethnic Origin, Political/Religious/Philosophical Beliefs or Opinions, Trade Union Membership, Genetic or Biometric Data (where this is used for identification purposes e.g. fingerprint or facial recognition), Physical and/or Mental Health Data, Sexual Orientation or Sex Life, Gender (self-declared or observed), Offences Committed/Alleged to have Committed/Criminal Proceedings/Outcomes/Sentence, Other Online Identifiers, Other (please specify). Logic DISABLEMENT LOGIC - Disabled if D01 is no OR if DS06 is yes. OR if DS08 does not contain physician/healthcare professional AND DS10 does not contain physician/healthcare professional. Scoring Impact There is no scoring impact associated with this question.
What data is automatically shared with other users?	ORC_DS14
Further Information Guidance/Context If the policy has clearly stated that data is automatically shared with other users but is not clear exactly what data is shared and assessors are unable tp infer this from using the app, it should be assumed that all collected data is automatically shared. Do not select Biometric data if it is using the phone's native scanner/face ID rather than an in-app one scanner. Response Multiple Choice Answer Criteria Name (Full Name, Nickname or First Name Only), Username, Mobile Number/Device Number/Home Phone Number, Email, Full Address/Postcode, Age/DOB, Location Data, IP Address, Device IMEI Number, Other Unique Device Identifiers, General Wellness Data, General Identifier e.g. NHS No, Physical Description, Martial Status/Family/Lifestyle/Social Circumstance, Income/Financial/Tax Situation, Education/Qualifications/Professional Training/Awards, Employment/Career History, Cookies/Web Beacons etc. (used for tracking an individual’s online browsing behaviors/movements), Usage Data, Card/Payment/Financial Information, Race/Ethnic Origin, Political/Religious/Philosophical Beliefs or Opinions, Trade Union Membership, Genetic or Biometric Data (where this is used for identification purposes e.g. fingerprint or facial recognition), Physical and/or Mental Health Data, Sexual Orientation or Sex Life, Gender (self-declared or observed), Offences Committed/Alleged to have Committed/Criminal Proceedings/Outcomes/Sentence, Other Online Identifiers, Other (please specify). Logic DISABLEMENT LOGIC - Disabled if D01 is no OR if DS06 is yes. OR if DS08 does not contain other users AND DS10 does not contain other users. Scoring Impact There is no scoring impact associated with this question.
What data is automatically shared with third parties?	ORC_DS15
Further Information Guidance/Context If the policy has clearly stated that data is automatically shared with third parties but is not clear exactly what data is shared then assessors can assume that all collected data is automatically shared. Do not select Biometric data if it is using the phone's native scanner/face ID rather than an in-app one scanner. Response Multiple Choice Answer Criteria Name (Full Name, Nickname or First Name Only), Username, Mobile Number/Device Number/Home Phone Number, Email, Full Address/Postcode, Age/DOB, Location Data, IP Address, Device IMEI Number, Other Unique Device Identifiers, General Wellness Data, General Identifier e.g. NHS No, Physical Description, Martial Status/Family/Lifestyle/Social Circumstance, Income/Financial/Tax Situation, Education/Qualifications/Professional Training/Awards, Employment/Career History, Cookies/Web Beacons etc. (used for tracking an individual’s online browsing behaviors/movements), Usage Data, Card/Payment/Financial Information, Race/Ethnic Origin, Political/Religious/Philosophical Beliefs or Opinions, Trade Union Membership, Genetic or Biometric Data (where this is used for identification purposes e.g. fingerprint or facial recognition), Physical and/or Mental Health Data, Sexual Orientation or Sex Life, Gender (self-declared or observed), Offences Committed/Alleged to have Committed/Criminal Proceedings/Outcomes/Sentence, Other Online Identifiers, Other (please specify). Logic DISABLEMENT LOGIC - Disabled if D01 is no OR if DS06 is yes. OR if DS08 does not contain third parties AND DS10 does not contain third parties. Scoring Impact There is no scoring impact associated with this question.
What data is automatically shared with other devices?	ORC_DS16
Further Information Guidance/Context If the policy has clearly stated that data is automatically shared with other devices but is not clear exactly what data is shared then assessors can assume that all collected data is automatically shared. Do not select Biometric data if it is using the phone's native scanner/face ID rather than an in-app one scanner. Response Multiple Choice Answer Criteria Name (Full Name, Nickname or First Name Only), Username, Mobile Number/Device Number/Home Phone Number, Email, Full Address/Postcode, Age/DOB, Location Data, IP Address, Device IMEI Number, Other Unique Device Identifiers, General Wellness Data, General Identifier e.g. NHS No, Physical Description, Martial Status/Family/Lifestyle/Social Circumstance, Income/Financial/Tax Situation, Education/Qualifications/Professional Training/Awards, Employment/Career History, Cookies/Web Beacons etc. (used for tracking an individual’s online browsing behaviors/movements), Usage Data, Card/Payment/Financial Information, Race/Ethnic Origin, Political/Religious/Philosophical Beliefs or Opinions, Trade Union Membership, Genetic or Biometric Data (where this is used for identification purposes e.g. fingerprint or facial recognition), Physical and/or Mental Health Data, Sexual Orientation or Sex Life, Gender (self-declared or observed), Offences Committed/Alleged to have Committed/Criminal Proceedings/Outcomes/Sentence, Other Online Identifiers, Other (please specify). Logic DISABLEMENT LOGIC - Disabled if D01 is no OR if DS06 is yes. OR if DS08 does not contain other devices AND DS10 does not contain other devices. Scoring Impact There is no scoring impact associated with this question.
Does the app allow users to access to their personal health record via an ONC certified EHR?	ATA_DS01
Further Information Guidance/Context ONC = Office of national coordinator The ONC Health IT Certification Program is meant to signal which health IT systems meet federal requirements and include useful functionality. Response Yes / No Answer Criteria Yes: If users are able to access their Patient Records/Data held in an ONC certified EHR through the app. No: If users cannot access Patient Records through the app. No: If users can create personal health records that are separate from ONC certified EHR. Logic DISABLEMENT LOGIC - Scoring Impact There is no scoring impact associated with this question.
Does the app appear to access and/or process patient information from an ONC certified EHR?	ATA_DS02
Further Information Guidance/Context The ONC Health IT Certification Program is meant to signal which health IT systems meet federal requirements and include useful functionality. Response Yes / No Assessment Criteria Yes: If the app appears to access and/or process EHR Patient Data in any way. Yes: If a Clinician/HCP focused app that has been designed to allow the HCP to access and view patient records that have been pulled into the product from EHR Patient Data or the data is exported and added to individual EHR Patient Data No: If the app allows users or HCPs to build individual user/patient records and does not integrate with EHR Patient Data in any way, but instead simply keeps a separate record. Logic DISABLEMENT LOGIC - Scoring Impact There is no scoring impact associated with this question.

Algorithm/AI

Question Set

Does the app contain algorithms?	ORC_AI01
Further Information Guidance/Context This question aims to identify if there are any algorithms used in the app. This question then influences if other questions around AI and Clinical Calculators are asked later on in the assessment. Response Yes / No Answer Criteria Yes: If the app uses an algorithm to provide an output, using the health data input by the user OR If the app provides an average from input data, or calories burned. No: If the app does not calculate anything with the data it collects OR If the algorithm doesn’t come from health data input. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
How does the app use the algorithm?	ORC_AI02
Further Information Guidance/Context This question allows the assessor to describe what the algorithm does and what area it focuses on. There are questions later on in scene setters which will probe diagnoses/treatment further. Response Free Text - E.g. perform a calculation for diagnosis etc. Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if AI01 is no. Scoring Impact There is no scoring impact associated with this question.
Does the app appear to use AI?	ORC_AI03
Further Information Guidance/Context This question aims to identify if is any form of AI used within the app. Somethings this may be difficult to decipher from just using the app so the assessor should read around the app to see if the Developer makes these claims. Response Yes/ No Answer Criteria YES: If the app uses a chatbot which learns from and reacts to what the user says. YES: If the app/developer claims to use AI techniques. YES: If the app uses machine learning to improve the quality of its automated decision making. NO: If the app doesn’t use a chatbot. NO: If the app/developer makes no claim about using AI techniques. NO: If the app doesn’t use machine learning to improve the quality of its automated decision making. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
What AI technique is used in the app?	ORC_AI04
Further Information Guidance/Context The Developer may state in app store description or their website what type of AI they use. If it is unclear what AI is used within the app, the assessor should try and find this information through reading the app store description and app website. Response Free text Answer Criteria Examples: Natural Language Processing (NLP) - Includes Natural Language Understanding, Natural Language generation, Machine Translation. E.g. If the app uses a chatbot which learns from and reacts to what the user says. Machine Learning - If the app uses machine learning to improve the quality of its automated decision making. Image Recognition - If the app uses AI to identify something in a picture Logic DISABLEMENT LOGIC - Disabled if AI03 is no. Scoring Impact There is no scoring impact associated with this question.
Is the AI monitored/ maintained?	ORC_AI05
Further Information Guidance/Context To ensure AI is used both appropriately and effectively, humans should have oversight through monitoring/maintaining/updating the app. Developers may monitor their AI by asking healthcare professionals to review the decision making and output. If the output does not appear be in line with the healthcare professionals knowledge, the Developer should correct this. Assessors should look for these mentions via the app, the app/google play store and associated website. Response Yes / No Answer Criteria YES: If the developer mentions specifically that their AI is monitored/maintained/updated. No: If there is no specific mention of them updating/maintaining the AI. No: If they only mention improvements based upon input (learning from input). Logic DISABLEMENT LOGIC - Disabled if AI03 is no. Scoring Impact There is no scoring impact associated with this question.

Information

Question Set

Is the app designed to provide information or guidance?	ORC_I01
Further Information Guidance/Context This question aims to contribute to the apps functions and features. It is also used to inform the ESF tiering. Response Yes / No Answer Criteria Yes: If the app provides any generic info around the general topic area, in text or diagram form. Yes: If the app can provide information as a diary back to the user if monitoring is taking place. No: If the app provides no real information or guidance aimed at health or wellbeing. No: If the only information is provided by other users on a forum, information must come from the developer/app itself. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Does the app provide information that is personalized to an end user’s specific circumstances?	ORC_I02
Further Information Guidance/Context This question aims to contribute to the apps functions and features. It is also used to inform the ESF tiering. Response Yes / No Answer Criteria Yes: If any of the information provided is personalized to the user. E.g. provides recommended activities/actions based on assessment over a period of time OR tailors therapy/treatment program based on one off assessment which includes taking a lot of information from the user. No: If the app provides no information which is personalized to the user. Logic DISABLEMENT LOGIC - Disabled if I01 is no. Scoring Impact There is no scoring impact associated with this question.
Does the app provide users with information regarding where they are able to find local or suitable support services?	ORC_F08
Further Information Guidance/Context This question aims to contribute to the apps functions and features. It is also used to inform the ESF tiering. Response Yes / No Answer Criteria YES: If the app provides links/signposts to online services or local services. YES: If the app points to services where the user can take control of their/somebody’s condition e.g. pharmacy. NO: If the app provides no information which is personalized to the user. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Does the app provide environmental data not specific to the patient?	ORC_F03
Further Information Guidance/Context This question aims to contribute to the apps functions and features. It is also used to inform the ESF tiering. Response Yes / No Answer Criteria YES: If the app provides details of external environmental factors which may impact health/wellbeing, such as temperature, pollen count etc. NO: If the only information provided is the location. Logic DISABLEMENT LOGIC - Disabled if I01 is no. Scoring Impact There is no scoring impact associated with this question.
Does the App provide information, resources or activities to the public, patients or physicians, either about a specific condition or general health and lifestyle?	ORC_EF07
Further Information Guidance/Context In most cases the answer to this question will be yes because the scope of the question is so broad. The only instance an assessor should answer no to this question is if the app is aimed at providing information, resources or activities for administrative purposes instead of health related purposes. If I01 is yes, this will also be yes. This question also guides the ESF tiering later. Response Yes / No Answer Criteria Yes: Any app that provides a resource, either condition specific or generalized. No: Administration apps which have no effect on patient outcomes for instance a schedule system. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.

Clinical Decision Support - Pre-Diagnosis, Diagnosis and Treatment Support

Question Set

Is the data the app collects, automatically assessed, for the purposes of evaluating: risk; or providing diagnostic support?	ORC_PD01
Further Information Guidance/Context This is looking at apps which provide individual risk to a user, which is personalized based on the user health data collected (e.g. apps which have red zones/percentages of having a condition for specific readings). Response Yes / No Answer Criteria Yes: If the app provides a calculation, paragraph, table or diagram indicating a users risk, or potential diagnoses. No: If the app provides no form of risk assessment or diagnoses to a user. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Does the app diagnose a specific condition?	ORC_DG02
Further Information Guidance/Context This question aims to discover if the app diagnoses/screens/detects a disease or condition (i.e., using sensors, data, or other information from other hardware or software devices, pertaining to a disease or condition). This is a key question for identifying diagnostic medical devices under FDA. Response Yes / No Answer Criteria Yes: A healthcare professional can see “We think you have..”. Yes: If the app diagnoses a specified clinical condition using clinical data. No: If it states ‘you might have a condition please see a professional’, this would be no as it is not specific enough. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Does the app provide an assessment (of the risk) to an individual - based on data input or collected by the app - of: Contracting or Suffering a healthcare condition The impact on their lifestyle and health indicators No Risk Assessment provided	ORC_DG01
Further Information Guidance/Context This is looking at apps which provide individual risk assessments to the user. Both ‘Contracting or suffering a healthcare condition’ and ‘The impact of their lifestyle choices and health and wellbeing indicated’ can be selected at the same time. Response Multiple Choice Answer Criteria Contracting or suffering a healthcare condition: The app provides a risk indicator of a condition or diagnosis to an individual. The impact of their lifestyle choices and health and wellbeing indicators: The app provides a risk indicator based on the person's lifestyle to an individual, eg a person’s risk of type 2 diabetes based on food intake. No risk assessment provided: The app provides no risk indicator or diagnoses. Logic DISABLEMENT LOGIC - Disabled if PD01 is no AND DG02 is no. Scoring Impact There is no scoring impact associated with this question.
Does the app provide an assessment (of the risk) to a healthcare professional - based on data input or collected by the app - of: Contracting or Suffering a healthcare condition The impact on their lifestyle and health indicators No Risk Assessment provided	ORC_DG03
Further Information Guidance/Context This is looking at apps which provide individual risk assessments to a healthcare professional. Both ‘Contracting or suffering a healthcare condition’ and ‘The impact of their lifestyle choices and health and wellbeing indicated’ can be selected at the same time. Response Multiple Choice Answer Criteria Contracting or suffering a healthcare condition: The app provides a risk indicator of a condition or diagnosis to an individual. The impact of their lifestyle choices and health and wellbeing indicators: The app provides a risk indicator based on the person's lifestyle to an individual, eg a person’s risk of type 2 diabetes based on food intake. No risk assessment provided: The app provides no risk indicator or diagnoses. Logic DISABLEMENT LOGIC - Disabled if PD01 is no AND DG02 is no. Scoring Impact There is no scoring impact associated with this question.
Does the app provide the option for further assessment or analysis by a healthcare professional?	ORC_DG04
Further Information Guidance/Context This question is applicable to apps which provide individual risk to a user using some kind of algorithm or AI. It aims to identify if the user can send their result/information to a healthcare professional to get a further assessment, for more information. Response Yes / No Answer Criteria Yes: If the app provides a calculation, paragraph, table or diagram indicating a users risk, and allows this to be sent to a HCP for further investigation/information (being able to get a second opinion from a real clinician by sending information on). No: If the app provides no form of further investigation/information by a HCP. No: If the app offers solely virtual consultations with HCP with no transfer of health data logged within the app. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Is the app/does the app include a Symptom Checker?	ORC_DG05
Further Information Guidance/Context This questions aims to identify apps which collect check the user’s symptoms and provide a possible diagnosis/diagnoses based on the inputted information. The purpose or benefit of the app must be its symptom checking functionality. For instance, if an anxiety/depression app contained GAD-7 or PHQ-9, this would not be a symptom checker. Response Yes / No Answer Criteria Yes: If the app provides a possible diagnosis based upon the collection of a user’s symptoms. No: If the app provides no form of diagnosis or risk assessment based upon the collection of a user’s symptoms. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Does the app indicate likelihood of a match for the listed conditions?	ORC_DG06
Further Information Guidance/Context This is looking at whether a symptom checker provides an assessment of what is the most likely cause to the user. For example, 9/10 people who have your symptoms suffer from X. Response Yes / No Answer Criteria Yes: If the app provides an assessment of chance, or likelihood of certain conditions based on collected symptoms. No: If the app provides no assessment of chance, or likelihood of certain conditions based on collected symptoms. Logic DISABLEMENT LOGIC - Disabled if DG05 is no. Scoring Impact There is no scoring impact associated with this question.
Can users filter results to display by highest risk / likelihood / severity?	ORC_DG07
Further Information Guidance/Context This question aims to identify if an app allows the user to filter through the list of conditions the symptom checker provided. It is crucial that the user has the option to turn the filter on or not, if the list is automatically generated in a particular order, this is not providing the user the autonomy to filter. Any filtering rules are applicable from likelihood of symptoms matching a condition to most severe condition symptoms may relate to. Response Yes / No Answer Criteria Yes: If the app provides a filter for the provided risks. The app needs to provide a specific filter option, and sorting by order of likelihood automatically is NOT sufficient. No: If the app provides no filter for the provided risks. Logic DISABLEMENT LOGIC - Disabled if DG05 is no. Scoring Impact There is no scoring impact associated with this question.
Does the app provide treatment recommendations for the listed conditions?	ORC_DG08
Further Information Guidance/Context This question aims to identify if symptom checker apps provide treatment suggestions alongside the listed conditions. If next to a symptom the app recommends a user should seek treatment through signposting to further services, this is not sufficient. The app must be providing the treatment details itself for this question. Response Yes / No Answer Criteria Yes: If the app provides any treatment suggestions for the listed conditions. No: If the app provides no treatment options for the listed conditions. Logic DISABLEMENT LOGIC - Disabled if DG05 is no. Scoring Impact There is no scoring impact associated with this question.
Does the app only signpost the user to suitable care or recommend seeking further advice? (eg. Go to ER, book an appointment with your family physician, call 911)	ORC_DG09
Further Information Guidance/Context This question aims to identify if the symptom checker app provides suggestions for the user to seek further treatment, based upon the indicated diagnoses. Anything from calling 911 to a recommended visit to your family physician would be sufficient for this question. Response Yes / No Answer Criteria Yes: If the app provides any signpost to a further service based upon the symptom checker outcome. No: If the app does not signpost to a further service based upon the symptom checker outcome. Logic DISABLEMENT LOGIC - Disabled if DG05 is no. Scoring Impact There is no scoring impact associated with this question.
Does the app contain a clinical calculator?	ORC_TS01
Further Information Guidance/Context The FDA takes into account the use of algorithms and AI in relation to identifying and assessing medical devices. The identification of a clinical calculator would help support the argument as to whether an app should be classified as a medical device or not. Response Yes / No Answer Criteria Yes: This includes apps for use by clinicians or users to calculate parameters pertaining to care, such as early warning system software (if there are treatment implications associated with the calculation). No: If the app is not for use by clinicians or users to calculate parameters pertaining to care, such as early warning system software. Logic DISABLEMENT LOGIC - Disabled if AI03 is no AND if AI01 is no. Scoring Impact There is no scoring impact associated with this question.
What type of clinical calculator does the app contain?	ORC_TS02
Further Information Guidance/Context This question allows the assessor to record information about the type of Clinical Calculator which has been located within the app. For example, contained within the app may be something which calculates the amount of water needed to treat a burns victim. Response Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if TS01 is no. Scoring Impact There is no scoring impact associated with this question.
Is the app intended to be (or does the developer claim it can be) used for the prevention of disease?	ORC_MD01
Further Information Guidance/Context Prevention is another key definition in defining a medical device according to FDA regs. If an app is claiming to be used for prevention or its intended use/benefit relates to prevention of a disease or condition it is likely to be a medical device. Response Yes / No Answer Criteria Yes: If the app is intended to PREVENT a specific disease or condition OR If the actual app will stop you from getting the disease. No: If the app is trying to catch something early before it develops, e.g. skin vision. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
How does the app prevent disease?	ORC_TS04
Further Information Guidance/Context This question allows the assessor to explain how the app prevents disease. Response Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if MD01 is no. Scoring Impact There is no scoring impact associated with this question.
Does the app provide treatment of a condition?	ORC_TS05
Further Information Guidance/Context This question aims to identify if an app provides treatment to a user’s specific condition. This includes both apps that provide information which can be used to enable treatment as well as apps which provide an output which can be used to treat a condition. For example, apps which calculate that are intended to calculate the dose of insulin a diabetic needs to treat their diabetes based on carbohydrate in a meal. Response Yes / No Answer Criteria Yes: Apps that provide information that can be used to enable treatment to be performed or claim that the output from the app can be used to treat a condition. E.g. an app to calculate the dose of insulin a diabetic needs to treat their diabetes based on carbohydrate in a meal. No: If the app is intended to treat non-medical conditions e.g. non-specific stress. OR apps intended to just provide tips or advice or link to support groups OR medication reminders. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
What treatment does the app provide?	ORC_TS06
Further Information Guidance/Context This question allows the assessor to explain what treatment the app provides. Response Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if TS05 is no. Scoring Impact There is no scoring impact associated with this question.
Does the app guide the treatment of a condition?	ORC_TS07
Further Information Guidance/Context This question aims to identify apps which are guiding the treatment of a condition. This can occur in a number of ways but it is key that the app is guiding the treatment of a condition following best clinical practice guidelines. Response Yes / No Answer Criteria Yes: Apps which take a user’s health information, and provide specific treatment pathways for the user to follow to treat their condition OR clinician-facing apps that advise treatments. No: Apps intended to just provide tips or advice which is non-specific to the user. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
How does the app guide the treatment of the condition?	ORC_TS08
Further Information Guidance/Context This question allows the assessor to explain how the app can guide the user’s treatment of a condition. Response Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if TS07 is no. Scoring Impact There is no scoring impact associated with this question.
Who does the app provide the treatment guidance to?	ORC_TS09
Further Information Guidance/Context The question allows the assessor to confirm whether treatment guidance is for a general user or a healthcare professional. Response Multiple Choice Answer Criteria User: Refers to the patient / carer using the app. HCP: Health Care Professional. Logic DISABLEMENT LOGIC - Disabled if TS07 is no. Scoring Impact There is no scoring impact associated with this question.
Is the treatment provided independently of a healthcare professional?	ORC_TS10
Further Information Guidance/Context This question aims to determine whether the app can provide treatment to an individual without a healthcare professional involvement. Response Yes / No Answer Criteria Yes: If the app provides treatment to the user without HCP involvement. No: If the treatment is not provided independently of a HCP, or if the app provides no treatment. Logic DISABLEMENT LOGIC - Disabled if TS05 is no AND if TS07 is no. Scoring Impact There is no scoring impact associated with this question.
Does the app support healthcare professionals’ decisions about treatments?	ORC_TS03
Further Information Guidance/Context This question aims to identify apps which supports a decision made by a healthcare professional on a case by case basis. The app must be more than a generic textbook and provide information directed towards healthcare professionals. Response Yes / No Answer Criteria Yes: The app contributes to a professional’s decision about treatment, so this is for doctors to look at OR Supports decision making on a case by case basis (eg Mersey Burns would be yes (tells clinician how much fluid a patient requires based on percentage burns they have suffered). No: If the app provides generic, non specific care pathways. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Does the app follow the path of a procedure/treatment without making any decisions?	ORC_TS12
Further Information Guidance/Context Whilst the FDA regulations do not specify if following a path of a procedure/treatment makes an app a medical device, the assessor will be made aware of the risk which comes with this functionality as well as understanding that additional functionality which could lead the app to be a medical device under FDA regulations. Response Yes / No Answer Criteria YES: If the app outlines a treatment / procedure but does not make and communicate any decisions to the user. NO: If the app makes any decisions for the user. Logic DISABLEMENT LOGIC - Disabled if TS03 is no. Scoring Impact There is no scoring impact associated with this question.
Does a healthcare professional make the final decision regarding treatment based on advice and/or options displayed?	ORC_TS13
Further Information Guidance/Context Apps could still potentially be a MD if this is answered as yes, relying on additional functionality too. FDA is unclear about specifics in this area (would depend on other functions too). That said, if this question is answered no, it is very likely the app would be identified as a MD under the FDA. Response Yes / No Answer Criteria YES: If the app outlines a treatment/ procedure but does not make and communicate any decisions to the user. NO: If the app makes any decisions for the user. Logic DISABLEMENT LOGIC - Disabled if TS03 is no. Scoring Impact There is no scoring impact associated with this question.
Does the app automate the treatment pathway for an individual patient?	ORC_TS14
Further Information Guidance/ Context Automating the treatment pathway is a software function that makes the app become a regulated medical device by performing patient-specific analysis and providing patient-specific diagnosis, or treatment recommendations. Response Yes / No Answer Criteria YES: The app creates the treatment pathway for the user, and does not rely on a HCP. NO: If the app outlines a set of treatments / procedures but the final decision about which treatment is left to the HCP. NO: If it is a “one size fits all” pathway that doesn’t take into account individual factors. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Is the app intended to be (or does the developer claim it can be) used as a physical intervention to reduce the symptoms or severity of a disease, injury or, physical or mental impairment?	ORC_TS15
Further Information Guidance/Context This question aims to identify if the apps intended purpose is to reduce the symptoms or severity of a disease, injury or physical/mental impairment. The app cannot just support a condition or impairment, the app must provide some sort of functionality to reduce the symptoms. For example, a Tinnitus noise cancelling app reduces the symptoms of Tinnitus. This is to help identify Medical Devices. Response Yes / No Answer Criteria Yes: If an app that provides a physical output to alleviate the symptoms of an existing condition. For example a Tinnitus noise canceling app. No: If an app does not provide a physical output to alleviate the symptoms of an existing condition. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Is the app intended to (or does the developer claim it can be used to) compensate an injury or, physical or mental impairment?	ORC_MD07
Further Information Guidance/Context This question aims to identify if an app compensates for a specific injury or physical / mental impairment. It is important that the assessor identifies this is the app's intended purpose and that it is not meant for general use. Response Yes / No Answer Criteria Yes: Apps which the developer claims can compensate for an injury or handicap or claims that the output from the app can be used for this purpose. For example apps to magnify text specifically for people with visual impairment or apps amplify sounds for people with reduced hearing. No: If the app provides no link to a specific injury or handicap. For example apps to magnify text but there is no mention of visual impairment in the manufacturer’s claims OR apps to amplify sound but there is no mention of hearing impairment in the manufacturer’s claims. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Does the app predict the fertile window?	ORC_CC01
Further Information Guidance/Context This question helps identify if the app’s intended purpose/benefits focuses around conception/contraception. This will in turn help identify if the app is a potential Medical Device as defined by the Food and Drug Administration (FDA). Response Yes / No Answer Criteria YES: If an app informs the user of when their fertile window is. NO: If the app does not identify the user’s fertile window. Logic DISABLEMENT LOGIC - Disabled if MD08 is no. Scoring Impact There is no scoring impact associated with this question.
Does the app claim to be used to prevent pregnancy or to conceive?	ORC_CC02
Further Information Guidance/Context This question helps identify if the app’s intended purpose/benefits focuses around conception/contraception. This will in turn help identify if the app is a potential Medical Device as defined by the Food and Drug Administration (FDA). Response Yes / No Answer Criteria YES: If an app or related website or description claims to make pregnancies more likely or to be able to prevent pregnancy. NO: If an app or related website or description does not claim to make pregnancies more likely or to be able to prevent pregnancy. Logic DISABLEMENT LOGIC - Disabled if CC01 is no. Scoring Impact There is no scoring impact associated with this question.
Does the app use body basal temperature (bbt) recorded through an externally connected thermometer?	ORC_CC03
Further Information Guidance/Context This question helps identify if the app’s intended purpose/benefits focuses around conception/contraception. This will in turn help identify if the app is a potential Medical Device as defined by the Food and Drug Administration (FDA). Response Yes / No Answer Criteria YES: If an app has an assistive device which can be used to record BBT. Measurements can be input manually after taking a reading. NO: If an app does not have an assistive device which can be used to record BBT., or it does not record BBT. Logic DISABLEMENT LOGIC - Disabled if CC02 is no. Scoring Impact There is no scoring impact associated with this question.
Does the app use rhythm, body basal temperature (bbt) and cervical mucus methods to prevent pregnancy or to conceive?	ORC_CC04
Further Information Guidance/Context This question helps identify if the app’s intended purpose/benefits focuses around conception/contraception. This will in turn help identify if the app is a potential Medical Device as defined by the Food and Drug Administration (FDA). Response Yes / No Answer Criteria YES: If an app uses rhythm/bbt/cervical mucus methods to aid in the prevention of pregnancy or to help conception. NO: If an app doesn’t use rhythm/bbt/cervical mucus methods to aid in the prevention of pregnancy or to help conception. Logic DISABLEMENT LOGIC - Disabled if CC02 is no. Scoring Impact There is no scoring impact associated with this question.
Does the developer claim that the app can be used as a natural method of birth control?	ORC_CC05
Further Information Guidance/Context This question helps identify if the app is marketed towards facilitating conception/contraception. This will in turn help identify if the app is a potential Medical Device as defined by the Food and Drug Administration (FDA). Response Yes / No Answer Criteria Yes: If the app markets itself or claims the user can use the app as a natural method of birth control. No: If the app does not claim to be a natural method of birth control. Logic DISABLEMENT LOGIC - Disabled if CC03 is no AND if CC04 is no. Scoring Impact There is no scoring impact associated with this question.
Is the app intended to be used for the control of conception?	ORC_MD06
Further Information Guidance/Context This question aims to identify if apps are to be used to control conception through two or more of the practical methods highlighted in CC01 - CC04. This will in turn help identify if the app is a potential Medical Device as defined by the Food and Drug Administration (FDA). Response Yes / No Assessment Criteria Yes: If the app appears to be a natural form of contraception AND be intended to be used a way of conceiving based on the above answers (CC01, CC02, CC03, CC04 and CC05). No: If the app only claims to be a natural form of contraception OR intended to be used as a way of conceiving, but doesn’t claim to be both. Logic DISABLEMENT LOGIC - Disabled if CC02 is no. Scoring Impact There is no scoring impact associated with this question.
Is the app used in combination with drugs or medication? (e.g. medication reminders)	ORC_AE20
Further Information Guidance/Context This question aims to identify if the app can set medication alerts/reminders, trackers or if the app indicates how much medication the user should take. Response Yes / No Answer Criteria Yes: If an app provides medication reminders or trackers used as an assistive tool OR if the app influences how much you should take e.g. insulin calculator etc. No: If an app does not provide medication reminders or trackers used as an assistive tool OR if there are no alarms to take medications. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Is the app a companion of the device, as opposed to having been designed to connect with a third party manufacturer's device?	ORC_F26
Further Information Guidance/Context This question aims to understand the context of the app and to also examine whether or not it can be used without the device. Response Yes / No Answer Criteria Yes: The app is designed to work with a specific device, and likely doesn’t function fully without it e.g. Garmin Watch with Garmin App. No: The app connects with third party devices e.g. Fitbit watch. Logic DISABLEMENT LOGIC - Disabled if DC01 does not contain Devices. Scoring Impact There is no scoring impact associated with this question.

Monitoring

Question Set

Does the app allow the monitoring of key health information?	ORC_MN01
Further Information Guidance/Context Due to a logic issue, only one answer should be selected. If the app allows the monitoring of both General Health or Wellness and Specific Condition Data, the assessor should select Specific Condition data. This question will contribute to the outcome of MN04, a question which helps determine what ESF tier the app belongs in. Response Multiple Choice Answer Criteria Yes - General health or Wellness data: If the app has any capability at all which allows the user to monitor any health information which the app records. Yes - Specific Condition data: If the app is aimed towards someone with a pre-existing condition e.g. chronic pain, diabetes etc. No - None: If an app does not collect or allow the monitoring of health information. Logic DISABLEMENT LOGIC - Disabled if D01 is no. OR disabled if DT10 does not contain neither Physical and/or Mental Health Data nor General Wellness Data. Scoring Impact There is no scoring impact associated with this question.
Does the app involve the recording of relevant data over time for the user to access and review (with no ‘intelligent’ manipulation of that data by the app)?	ORC_MN02
Further Information Guidance/Context This question aims to identify if the app allows users to record health information which can be reviewed at a later date. This question contributes to the outcome of MN04, a question which helps determine what ESF tier the app belongs in. Response Yes / No Answer Criteria Yes: If an app allows the recording and reviewing of data over a period of time to allow the user to monitor their health information. No: If an app does not collect or allow the monitoring of health information. Logic DISABLEMENT LOGIC - Disabled if MN01 contains None. Scoring Impact There is no scoring impact associated with this question.
Does the app involve the automated assessment or interpretation of relevant data to deliver alerts, insights, reminders or adjustments regarding the user’s health or lifestyle?	ORC_MN03
Further Information Guidance/Context This questions aims to identify is an app provides further insight around the user’s health data it collects. The app needs to be providing novel insights, automated alerts or reminders from the user’s health data. If the health data is relayed back to the user with no additional information, MN03 will be no. This question contributes to the outcome of MN04, a question which helps determine what ESF tier the app belongs in. Response Yes / No Answer Criteria Yes: If an app allows the user to record health data, and then the app provides insight back to the user. For example in the form of alerts, reminders or adjustments regarding the user’s health/lifestyle. No: If an app doesn’t collect health data, or if it collects it and regurgitates it back to the user in the form of a simple graph, without any further insight or information. Logic DISABLEMENT LOGIC - Disabled if MN01 contains None. Scoring Impact There is no scoring impact associated with this question.
Is the app:	ORC_MN04
Further Information Guidance/Context This questions aims to differentiate between the different types of self-management and therefore different tiers from the Evidence Standards Framework. MN01, MN02 and MN03 all feed into the outcome of this question. Below is a diagram assessors refer to during the assessment process in order to decipher what type of self management tool they reviewing. Response Multiple Option Answer Criteria A Simple Self Management app: If an app is simple monitoring with wellbeing and general health focus = Tier Bi A Standard Self Management app: If an app is simple monitoring with a specific condition focus OR complex monitoring with a wellbeing and general health focus = Tier Bii A Complex Self Management app: If an app is complex monitoring with a specific condition focus = Tier C Logic DISABLEMENT LOGIC - Disabled if MN01 contains None. Scoring Impact There is no scoring impact associated with this question.
Is the output of the app’s monitoring intended to affect the treatment of an individual?	ORC_MN05
Further Information Guidance/Context This questions aims to evidence whether the monitoring the app performs is specifically there to impact on the user’s treatment. Response Yes / No Answer Criteria Yes: If the app provides a calculated output which is based on the health information collected, which may directly impact an individual’s decision regarding the treatment management of a condition. For example, a peak flow meter which shows decreasing measurements which acts as an early warning software. No: If the app is not intended to affect the treatment management and is only carrying out complex monitoring that may display trends or other interesting data points. Logic DISABLEMENT LOGIC - Disabled if MN04 does not contain Complex Self Management app. Scoring Impact There is no scoring impact associated with this question.
Does the app allow others (i.e. not the user) to monitor or view the health data captured?	ORC_MN06
Further Information This question aims to evidence whether the app allows the monitoring of the user’s health data by people who are not the user. Response Yes / No Answer Criteria Yes: If the app provides functionality within it that allows someone to monitor the user’s collected health data. This may be a HCP or may be a family member or friend. No: No functionality within the App for someone who is not the user to view the data collected within the app. Functionality needs to be within the app. Scoring DISABLEMENT LOGIC - Disabled if MN01 contains None. Logic There is no scoring impact associated with this question.
Does the app automatically measure and/or record data about a user’s specified condition, and transmit the data to a professional, caregiver or third party organization, without any input from the user?	ORC_MN07
Further Information Guidance/Context This question aims to identify whether the app automatically collects and sends health data, without any sort of intervention from the user. Response Yes / No Answer Criteria Yes: If the app collects user health data and transmits it to somebody else, without any sort of user intervention. No: The app does not automatically collect data OR the app does not automatically transmit data. Logic DISABLEMENT LOGIC - Disabled if MN04 does not contain Specific Condition data. Scoring Impact There is no scoring impact associated with this question.
Does the app generate any alarms or alerts from the data recorded by the app or a connected device?	ORC_MN08
Further Information Guidance/Context This question aims to look at whether the app sends a notification to the user/carer/healthcare professional based on any of the data recorded through the app itself or a connected device. For example, a diabetes app could automatically alert the user by creating a noise that notifies the user their blood glucose levels are either hypo (low) or hyper (high) in regards to the satisfactory levels they should usually be. Response Yes / No Answer Criteria Yes: If the app sends an alarm, alert or notification based on any of the data collected by the app itself or a connected device. No: The app does not generate alarms based on the health data input and the user has to set them themselves. Logic DISABLEMENT LOGIC - Disabled if MN06 is no. Scoring Impact There is no scoring impact associated with this question.
Are the alarms generated by user-defined filtering rules?	ORC_MN09
Further Information Guidance/Context This question aims to identify if the user can define the filtering rules surrounding the health data and choose what boundaries trigger an alarm / alert. Response Yes / No Answer Criteria Yes: If the app alerts the user or HCP to a predefined abnormality manually set by the user. No: The app does not generate alarms based on the health data input. No: The app does generate alarms based on health data input but the user can not set these parameters themselves. Logic DISABLEMENT LOGIC - Disabled if MN08 is no. Scoring Impact There is no scoring impact associated with this question.
What type of intervention or treatment does the app provide?	ORC_TS11
Further Information Guidance/Context This is to determine what type of treatment the app supplies. Assessors can select more than one, for example if an app had a diary and an insulin calculator both Self-management and Monitoring should be selected. This question helps the assessor place the app in the correct ESF tier. Response Multiple Choice Answer Criteria Preventative behavior change: If the app is intended to modify the users behavior to reduce the risk of a condition. Psychological intervention: If the app is intended to provide a psychological intervention to someone with a diagnosed psychological condition e.g. not non-specific stress. CBT: If the app is intended to provide Cognitive Behavioral Therapy to a user in full. Fertility: If the app is intended to be used to help with fertility treatments. Self-management (administering measures): If the app is intended to be used to help provide information about how much medicine should be taken e.g. diabetic patient advised to take X units of insulin based on information inputted into the app. Tailored treatment plan: If the app provides the user with a tailored treatment plan to improve their condition based on collected information. Monitoring (basic): E.g. diary. Logic DISABLEMENT LOGIC - Disabled if TS05 is no AND if TS07 is no AND MN01 is None. Scoring Impact There is no scoring impact associated with this question.

Online Consultations

Question Set

Can the app be used for patients to have online consultations, conversations, or related Health Care services with a healthcare professional?	ORC_F14
Further Information Guidance/Response This question provides further context around the functionality of the app. Response Yes / No Answer Criteria Yes: If the app allows the user to access consultation with relevant professionals. This would be a call or online chat directly with a doctor or professional in the relevant field. No: The app does not allow consultations or other communication with a relevant professional through the app. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Is this through video consultation?	ORC_OC02
Further Information Guidance/Context This question follows on from the previous questions and it is also a data capture question. It aims to determine how exactly online conversations and consultations are held. Response Yes / No Answer Criteria Yes: If the user can have a video consultation directly with a relevant profession via the app. No: If a video call consultation with the relevant professional is not available via the app. Logic DISABLEMENT LOGIC - Disabled if F14 is no. Scoring Impact There is no scoring impact associated with this question.
Does the app allow healthcare professionals to provide clinical advice, as opposed to the app providing advice itself?	ORC_EF09
Further Information Guidance/Context This question provides further context around the functionality of the app and importantly informs the user where the advice comes from. Response Yes / No Answer Criteria YES: If the app enables a HCP to provide advice in whatever format through the app. This may be video consultation, instant messaging or other platform communications. NO: The app does not allow consultations or other communication from a relevant professional through the app. Logic DISABLEMENT LOGIC - Disabled if F14 is no. Scoring Impact There is no scoring impact associated with this question.
If the app allows healthcare professionals to provide clinical advice through the app, rather than the app providing the advice itself, how does it do this?	ORC_OC01
Further Information Guidance/Context This question is a data capture question. It aims to collect information about exactly how the app allows a professional to supply clinical advice. Response Free text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if EF09 is no. Scoring Impact There is no scoring impact associated with this question.

Administrative Services

Question Set

Is this an administrative app which does not directly impact patient care?	ORC_AS01
Further Information Guidance/Context This question allows us to understand further the functionality of the app, it helps us with our adapted ESF tiering. Response Yes / No Answer Criteria Yes: If the app provides a digital solution for internal admin running of healthcare systems. If the app aids appointment bookings, staff roster, job lists etc. No: If there is any potential impact on a patients treatment, this includes messaging apps. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
What administrative functions does the app provide?	ORC_AS02
Further Information Guidance/Context This question allows us to understand further the functionality of the app, it helps us with our adapted ESF tiering. Response Multiple Choice Answer Criteria Schedule Management Appointment Booking Prescription Management Building Maintenance Logic DISABLEMENT LOGIC - Disabled if AS01 is no. Scoring Impact There is no scoring impact associated with this question.
Is the app used to facilitate communication between healthcare professionals other than for consultation or the delivery of advice?	ORC_AS03
Further Information Guidance/Context This question allows us to understand further the functionality of the app, it helps us with our adapted ESF tiering. Response Yes / No Answer Criteria Yes: If the app supports communication between healthcare professionals. No: If any communication within the app is between users or patients and does not involve a healthcare professional. Logic DISABLEMENT LOGIC - Disabled if AS01 is no. Scoring Impact There is no scoring impact associated with this question.
Does the app allow users to book appointments with a healthcare professional?	ORC_AS04
Further Information Guidance/Context This question allows us to understand further the functionality of the app. Response Yes / No Answer Criteria Yes: If users can book appointments with their own GP through patient access. Yes: If users can book appointments through the app. No: If you can only add appointments to a calendar for organizational purposes rather than actually booking in with a healthcare professional. Logic DISABLEMENT LOGIC - Disabled if AS01 is no. Scoring Impact There is no scoring impact associated with this question.

Pharmacy

Question Set

Does the app allow users to order and request prescriptions?

ORC_F13

Further Information

Guidance/Context

This question helps identify if an app can help a user order / request prescriptions.

Response

Yes / No

Answer Criteria

Yes: If the app allows users to order or request a prescription from a healthcare professional, healthcare provider or pharmacy.

No: If the app allows the user to record what prescription they would like to request. This would only be acting as a reminder to the user.

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

Reminders/Notifications

Question Set

Does the app send push notifications?	ORC_D29
Further Information Guidance/Context This is an information capture question which helps inform the functions and features. Response Yes / No Answer Criteria Yes: If the app sends push notifications to the device. No: If there are in-app notifications which are not pushed to the device. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Does the app send email notifications?	ORC_D30
Further Information Guidance/Context This is an information capture question which helps inform the functions and features. Response Yes / No Answer Criteria Yes: If the app sends email notifications relating to the user’s use of the app, personalized. No: If the only emails are marketing/newsletters. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.

External Device

Question Set

Is the app's main functionality dependent on the user having one of the devices to connect with the app?	ORC_F27
Further Information Guidance/Context This question aims to identify if an app can only be used for its intended purposes if a user has access to one of the connected devices. Response Yes / No Answer Criteria Yes: If the app ONLY works with a companion device. For instance, the user is unable to input data and therefore cannot use the app at all without the device. No: If there is a companion device but the app can still be used independently. Logic DISABLEMENT LOGIC - Disabled if DC01 does not contain Devices. Scoring Impact There is no scoring impact associated with this question.
Do any of the features or functions of the app appear to allow it to be used to control a medical device?	ORC_F30
Further Information Guidance/Context This question aims to Response Yes / No Answer Criteria Yes: If the app is used to control an external medical device. No: The app connects with an external device which is not classified as a medical device. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.

Forums and Contacts

Question Set

Are there opportunities to link with other users (buddying, forums or group education)?	ORC_U19
Further Information Guidance/Context This is an information capture question which helps inform the functions and features. Response Yes / No Answer Criteria Yes: If there is any way for users to communicate with other users within the app. This can be through messaging, internal forums, connecting with friends, communicating with a healthcare professional etc. No: If you can only send a report to a doctor via email for example. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Does the app provide an internally hosted forum or online community for their users?	ORC_FC01
Further Information Guidance/Context This question refers to forums which are within the app rather than ones hosted externally via Facebook, developer website etc. Response Yes / No Answer Criteria Yes: If the app has an internal forum. No: If the app provides links to a third party forum or an externally hosted forum. One-to-one communication is not a forum. Logic DISABLEMENT LOGIC - Disabled if U19 is no. Scoring Impact There is no scoring impact associated with this question.
Does the app link to a third-party service to host a forum or online community for their users?	ORC_FC02
Further Information Guidance/Context This question refers to forums which are hosted eternally via Facebook, developer website etc. rather than within the app. Response Yes / No Answer Criteria Yes: If the app provides links to a third party forum or an externally hosted forum. No: If the app links to a Facebook page which is not a forum. If the only forum is in-app. Logic DISABLEMENT LOGIC - Disabled if U19 is no. Scoring Impact There is no scoring impact associated with this question.
Does the app allow two-way communication between citizens, patients or healthcare professionals?	ORC_EF10
Further Information Guidance/Context This question is an information capture question which informs functions and features. The two way communication needs to exist within the app through chat functions, a forum, video call and must be between two or more people. Response Yes / No Answer Criteria Yes: If the app allows for any two-way communication between any two people. No: If the app does not enable two-way communication between two or more people. Logic DISABLEMENT LOGIC - Disabled if U19 is no. Scoring Impact There is no scoring impact associated with this question.

Goal Setting

Question Set

Does the app provide gamification or goal setting features for the user?	ORC_F06
Further Information Guidance/Context This question is an information capture question which informs functions and features. The gamification or goal setting features must somehow relate to the user’s health or wellbeing. Response Yes / No Answer Criteria Yes: If you can choose a goal, get badges or achievements through use of the app. Yes: If the app provides targets or you can set your own targets. Yes: If the app encourages engagement with rewards. No: No goal setting or gamification. If gamification has no real purpose. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Does the app set goals for the user?	ORC_GS01
Further Information Guidance/Context This question follows on from the previous and aims to identify what type of goals exist in the app. Response Multiple Choice Answer Criteria Tailored: If the goals are specific to the user. For example, the user can input health parameters and the app generates a goal based on those readings. Generic: If the set goals are generic for all users. For instance, goals that are pre-set within the app and are the same for each user. User defined: If the user can manually or directly specify or customize their goal. For example, the user can choose a weight loss goal which they can set themselves. Logic DISABLEMENT LOGIC - Disabled if F06 is no. Scoring Impact There is no scoring impact associated with this question.
Does the app allow the user to set goals for themselves?	ORC_U21
Further Information Guidance/Context This question allows us to understand further the functionality of the app, it helps us with our adapted ESF tiering. Response Yes / No Answer Criteria Yes: If ‘User defined' has been selected in the previous question (ORC_F06). No: If only ‘Tailored’ or ‘Generic’ has been selected in the previous question (ORC_F06). Logic DISABLEMENT LOGIC - Disabled if F06 is no. Scoring Impact There is no scoring impact associated with this question.

Customization

Question Set

Can the app presentation be customized by the user?	ORC_CUS01
Further Information Guidance/Context This question aims to identify if the user can edit the style of the app to suit their needs and / or preferences. Response Yes / No Answer Criteria Yes: If any changes can be made to the presentation theme within the app. This includes editing the background, colors, profile picture, language, measuring units etc. No: If the presentation of the app cannot be edited or customized by the user in any way. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Does the app respond to preferences in the device?	ORC_CUS02
Further Information Guidance/Context This question aims to identify if the user can set preferences on the device which is carried through to the app to suit their needs and / or preferences. Response Yes / No Answer Criteria Yes: If the app responds to changes in font size. Yes: If the app provides support options for users with poor vision/poor hearing. No: If the app only responds to in-app preferences. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.

Business Model

Question Set

Is the app totally free?	ORC_U29
Further Information Guidance/Context This is an information capture question which used to inform the users whether the app is completely free or if there are some sort of costs involved in the app. If the app has any costs associated with it ranging from in app purchases to licenses required by a healthcare provider, the assessor should answer this question no. If the app requires the user to purchase the associated device in order to use the app, the assessor should answer this question no. Response Yes / No Answer Criteria Yes: If the app is free to download AND has no in-app purchases or subscriptions AND costs are not incurred or covered by any third party organization/employees. For instance, licenses are NOT needed to be purchased for distribution). No: If licenses are needed to be purchased for distribution. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
How is the app funded?	ORC_BM01
Further Information Guidance/Context This question aims to identify what the business model is behind the app. If the answer is not apparent through publicly available information, the assessor should select Self-funded as a default. Response Multiple Choice Answer Criteria In-app purchase: If the app is funded through the user purchasing something within the app after downloading. Subscription: If the app is funded through subscription fees which the user has to pay in order to download the app. One off payment: If the app is funded through one off payments which the user needs pay in order to download the app. Licensed by doctor/healthcare provider: If the app is funded through licenses which need to be purchased in order for doctors/healthcare providers to provide access to their patients. Donations: If the app is funded through donations. Government or similar grant: If the app is funded through a government or similar grant. Charity / Non profit: If the app is funded by a charity or non-profit organization. Self-funded: If the app is self-funded by the people who run the company. OR if there is no evidence of how the app is funded then Self funded should be selected. Logic DISABLEMENT LOGIC - Disabled if U29 is no. Scoring Impact There is no scoring impact associated with this question.
Does the app contain advertisements?	ORC_U27
Further Information Guidance/Context This question aims to identify if the app displays advertisements for external products/services. If the app advertises their own subscriptions or in app purchases, this does not count. Response Yes / No Answer Criteria Yes: If the app has adverts for other products/services within it. No: If the app contains adverts for its own in-app purchases. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.

Benefits

Question Set

What are the claimed or implied benefits of the App?

ORC_BF01

Further Information

Guidance/Context

This questions aims to identify the intended purpose of the app through highlighting the claimed / implied benefits. If the assessor reads a clear benefit described by the developer, this would be a claimed benefit. If the assessor is having to infer a benefit from text written by the developer, this would be an implied benefit. If a claimed or implied benefit does not appear in the list below, the assessor should select Other Claimed/Implied Benefit and make the benefit very clear. In order for evidence to meet the requirements of the framework, the evidence of efficacy should relate to the benefits / intended purposes of the app.

Response

Multiple Choice

Answer Criteria

Cost savings to the healthcare system

Increased access to care

Improved diagnostic or risk assessment

Improved quality of treatment

Improved recovery

Reduced readmission or re-referral

Improved management of a condition

Preventative Behavior Change

Improved mental wellbeing

Improved physical wellbeing

Improved system/process efficiency

Other Claimed Benefit (please describe)

Other Implied Benefit (please describe)

Logic

There is no disablement logic written for this question.

Scoring Impact

There is no scoring impact associated with this question.

DATA & PRIVACY

Prior to answering any questions in the Data & Privacy area of the assessment, the Scene Setters will have captured much of the practical information about the observed data capture and use. There are no scoring implications of the Scene Setter questions. At this point, the assessment will have determined if any data is collected and retained, which data types are collected and shared, and how that data is used.
The DHA is particularly interested in whether the app collects personally identifiable data, or sensitive data, as well as cookies and device information data. If the app does make use of cookies, the DHAF will consider information also provided within the cookie policy (if available).
Also within the Scene Setters section, the DHAF looks at what user data is shared, who it is shared with, how it is shared (either manually or automatically), and whether the user has control or choice over this. The DHAF considers whether the app is able to connect to any third-party apps or external devices. If so, it is then considered whether the app offers the user any choice in connecting to other apps or devices. Data sharing to other apps or devices can be of benefit, providing the user has given explicit consent and has control over the sharing of their data.
The DHAF looks into data use, data storage and transit, data standards and management and compliance with HIPPA, and the application of the best practice as enshrined in the UK/EU General Data Protection Regulation 2018 (GDPR). The assessment looks into privacy information that is publicly available to the end-user, contained within the privacy policy applicable to the health app. The following questions provide detail of what information is expected to be provided to the user in relation to the use of their data.

Need for HIPAA Compliance

Click here to expand...

Does the person, business, or agency furnish, bill, or receive payment for health care in the normal course of business?	HIPAA1_1
Further Information Guidance/Context A provider of health care services and any other person or organization that furnishes, bills, or is paid for health care in the normal course of business. Response Yes / No Answer Criteria Yes: If an organisation conducting business in the US furnishes, bills or receives payment for the provision of health care to US citizens, then they are likely fall in scope of being a covered entity under HIPAA. No: The person, business, or agency is NOT a covered health care provider and therefore not a covered entity. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Does the person, business, or agency transmit (send) any covered transactions electronically?	HIPAA1_2
Further Information Guidance/Context Health care providers (e.g., physicians, hospitals, and clinics) are covered entities if they transmit health information in electronic form in connection with a transaction for which a HIPAA standard has been adopted by HHS. (e.g., billing) Response Yes / No Answer Criteria Yes: The person, business, or agency is a covered health care provider and therefore a covered entity. No: The person, business, or agency is NOT a covered health care provider and therefore not a covered entity. Logic DISABLEMENT LOGIC - Disabled if HIPAA1_1 is no. Scoring Impact There is no scoring impact associated with this question.
Does the business or agency process, or facilitate the processing of, health information from nonstandard format or content into standard format or content or from standard format or content into nonstandard format or content?	HIPAA2_1
Further Information Guidance/Context A public or private entity, including a billing service, repricing company, or community health information system, that processes non-standard data or transactions received from another entity into standard transactions or data elements, or vice versa. Response Yes / No Answer Criteria Yes: The business or agency may be a health care clearinghouse and therefore may be a covered entity No: The business or agency is NOT a health care clearinghouse and therefore not a covered entity. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Does the business or agency perform this function for another legal entity?	HIPAA2_2
Further Information Guidance/Context A public or private entity, including a billing service, repricing company, or community health information system, that processes non-standard data or transactions received from another entity into standard transactions or data elements, or vice versa. Response Yes / No Answer Criteria Yes: The business or agency is a health care clearinghouse and therefore a covered entity. No: The business or agency is NOT a health care clearinghouse and therefore not a covered entity. Logic DISABLEMENT LOGIC - Disabled if HIPAA2_1 is no. Scoring Impact There is no scoring impact associated with this question.
Is the plan an individual or group plan, or combination thereof, that provides, or pays for the cost of, medical care?	HIPAA3_1
Further Information Guidance/Context An individual or group plan that provides, or pays the cost of, medical care. Health plans include private entities (e.g., health insurers and managed care organizations) Response Yes / No Answer Criteria Yes: The plan may be a health plan and therefore a covered entity. No: The plan is NOT a health plan and therefore not a covered entity. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Is the plan a group health plan?	HIPAA3_2
Further Information Guidance/Context A “Group Health Plan” (GHP) is health insurance offered by an employer, union or association to its members while they are still working. GHP coverage is based on current employment. Response Yes / No Answer Criteria Yes: The person, business or agency may be a group health plan and therefore may be a covered entity. No: The person, business, or agency is NOT a group health plan and therefore not a covered entity. Logic DISABLEMENT LOGIC - Disabled if HIPAA3_1 is no. Scoring Impact There is no scoring impact associated with this question.
Does the plan have fewer than 50 participants?	HIPAA3_3
Further Information Active if HIPAA3_2 is YES Guidance/Context If answered no and the plan has more than 50 participants then this would not be a covered entity. Response Yes / No Answer Criteria Yes: The person, business or agency may be a group health plan and therefore may be a covered entity. No: The person, business, or agency is NOT a group health plan and therefore not a covered entity. Logic DISABLEMENT LOGIC - Disabled if HIPAA3_2 is no. Scoring Impact There is no scoring impact associated with this question.
Is the plan self-administered?	HIPAA3_4
Further Information Guidance/Context A self-insured group health plan (or a 'self-funded' plan as it is also called) is one in which the employer assumes the financial risk for providing health care benefits to its employees. Response Yes / No Answer Criteria Yes: The plan is NOT a health plan and therefore not a covered entity. No: The plan is a health plan and therefore a covered entity. Logic DISABLEMENT LOGIC - Disabled if HIPAA3_2 is no OR if HIPAA3_3 is yes. Scoring Impact There is no scoring impact associated with this question.
Is the plan a health insurance issuer?	HIPAA3_5
Further Information Guidance/Context Health insurance issuer (as defined in section 2791(b)(2) of the PHS Act, 42 U.S.C. 300gg-91(b)(2) means an insurance company, insurance service, or insurance organization (including an HMO) that is licensed to engage in the business of insurance in a State and is subject to State law that regulates insurance. Response Yes / No Answer Criteria Yes: The plan is a health plan and therefore a covered entity. No: The plan is not a health plan and therefore not a covered entity. Logic DISABLEMENT LOGIC - Disabled if HIPAA3_2 is yes. Scoring Impact There is no scoring impact associated with this question.
Is the plan an issuer of a Medicare supplemental policy?	HIPAA3_6
Further Information Guidance/Context Medigap is Medicare Supplement Insurance that helps fill "gaps" in Original Medicare and is sold by private companies. Original Medicare pays for much, but not all, of the cost for covered health care services and supplies. A Medicare Supplement Insurance (Medigap) policy can help pay some of the remaining health care costs, like: Copayments Coinsurance Deductibles Response Yes / No Answer Criteria Yes: The plan is a health plan and therefore a covered entity. No: The plan is not a health plan and therefore not a covered entity Logic DISABLEMENT LOGIC - Disabled if HIPAA3_5 is yes. Scoring Impact There is no scoring impact associated with this question.
Is the plan a health maintenance organization (HMO)?	HIPAA3_7
Further Information Guidance/Context HMO means "Health Maintenance Organization." HMO plans offer a wide range of health care services through a network of providers that contract exclusively with the HMO, or who agree to provide services to members at a pre-negotiated rate. As a member of an HMO, you will need to choose a primary care physician ("PCP") who will provide most of your health care and refer you to HMO specialists as needed. Some HMO plans require that you fulfill a deductible before services are covered. Others only require you to make a copayment when services are rendered. Health care services obtained outside of the HMO are typically not covered, though there may be exceptions in the case of an emergency. Response Yes / No Answer Criteria Yes: The plan is a health plan and therefore a covered entity. No: The plan is not a health plan and therefore not a covered entity. Logic DISABLEMENT LOGIC - Disabled if HIPAA3_6 is yes. Scoring Impact There is no scoring impact associated with this question.
Is the plan a multi-employer welfare benefit plan?	HIPAA3_8
Further Information Guidance/Context An arrangement offered by two or more employers to provide health or welfare benefits to the employers' employees and their beneficiaries, but excluding arrangements established or maintained: Under a collective bargaining agreement (CBA). Response Yes / No Answer Criteria Yes: The plan is a health plan and therefore a covered entity. No: The plan is not a health plan and therefore not a covered entity. Logic DISABLEMENT LOGIC - Disabled if HIPAA3_7 is yes. Scoring Impact There is no scoring impact associated with this question.
Is the plan an issuer of long-term care policies?	HIPAA3_9
Further Information Guidance/Context Under HIPAA, qualified long-term care services mean necessary diagnostic, preventive, therapeutic, curing, treating, mitigating, rehabilitative services, and personal care which are required by a chronically ill person and are provided according to a plan of care prescribed by a licensed health care practitioner. Response Yes / No Answer Criteria Yes: The plan may be a health plan and may be a covered entity. No: The plan is not a health plan and is therefore not a covered entity. Logic DISABLEMENT LOGIC - Disabled if HIPAA3_8 is yes. Scoring Impact There is no scoring impact associated with this question.
Does the plan provide only nursing home fixed-indemnity policies?	HIPAA3_10
Further Information Guidance/Context Fixed indemnity health insurance is a type of medical insurance that pays a predetermined amount on a per-period or per-incident basis, regardless of the total charges incurred. Response Yes / No Answer Criteria Yes: The plan is NOT a health plan and therefore not a covered entity. No: The plan is a health plan and therefore a covered entity. Logic DISABLEMENT LOGIC - Disabled if HIPAA3_8 is no. Scoring Impact There is no scoring impact associated with this question.
Does the plan provide only expected benefits?	HIPAA3_11
Further Information Guidance/Context The requirements of this part do not apply to any individual coverage in relation to its provision of the benefits described in paragraphs (a) and (b) of this section (or any combination of the benefits). (a) Benefits excepted in all circumstances. The following benefits are excepted in all circumstances: (1) Coverage only for accident (including accidental death and dismemberment) (2) Disability income insurance. (3) Liability insurance, including general liability insurance and automobile liability insurance. (4) Coverage issued as a supplement to liability insurance. (5) Workers' compensation or similar insurance. (6) Automobile medical payment insurance. (7) Credit-only insurance (for example, mortgage insurance). (8) Coverage for on-site medical clinics. (9) Travel insurance, within the meaning of § 144.103 of this subchapter. (b) Other excepted benefits. The requirements of this part do not apply to individual health insurance coverage described in paragraphs (b)(1) through (b)(6) of this section if the benefits are provided under a separate policy, certificate, or contract of insurance. These benefits include the following: (1) Limited scope dental or vision benefits. These benefits are dental or vision benefits that are limited in scope to a narrow range or type of benefits that are generally excluded from benefit packages that combine hospital, medical, and surgical benefits. (2) Long-term care benefits. These benefits are benefits that are either - (i) Subject to State long-term care insurance laws; (ii) For qualified long-term care insurance services, as defined in section 7702B(c)(1) of the Code, or provided under a qualified long-term care insurance contract, as defined in section 7702B(b) of the Code; or (iii) Based on cognitive impairment or a loss of functional capacity that is expected to be chronic. (3) Coverage only for a specified disease or illness (for example, cancer policies) if the policies meet the requirements of § 146.145(b)(4)(ii)(B) and (C) of this subchapter regarding non-coordination of benefits. (4) Hospital indemnity or other fixed indemnity insurance only if - (i) The benefits are provided only to individuals who attest, in their fixed indemnity insurance application, that they have other health coverage that is minimum essential coverage within the meaning of section 5000A(f) of the Internal Revenue Code, or that they are treated as having minimum essential coverage due to their status as a bona fide resident of any possession of the United States pursuant to Code section 5000A(f)(4)(B). (ii) There is no coordination between the provision of benefits and an exclusion of benefits under any other health coverage. (iii) The benefits are paid in a fixed dollar amount per period of hospitalization or illness and/or per service (for example, $100/day or $50/visit) regardless of the amount of expenses incurred and without regard to the amount of benefits provided with respect to the event or service under any other health coverage. (iv) A notice is displayed prominently in the application materials in at least 14 point type that has the following language: “THIS IS A SUPPLEMENT TO HEALTH INSURANCE AND IS NOT A SUBSTITUTE FOR MAJOR MEDICAL COVERAGE. LACK OF MAJOR MEDICAL COVERAGE (OR OTHER MINIMUM ESSENTIAL COVERAGE) MAY RESULT IN AN ADDITIONAL PAYMENT WITH YOUR TAXES.” (v) The requirement of paragraph (b)(4)(iv) of this section applies to all hospital or other fixed indemnity insurance policy years beginning on or after January 1, 2015, and the requirement of paragraph (b)(4)(i) of this section applies to hospital or other fixed indemnity insurance policies issued on or after January 1, 2015, and to hospital or other fixed indemnity policies issued before that date, upon their first renewal occurring on or after October 1, 2016. (5) Medicare supplemental health insurance (as defined under section 1882(g)(1) of the Social Security Act. 42 U.S.C. 1395ss, also known as Medigap or MedSupp insurance). The requirements of this part 148 (including genetic nondiscrimination requirements), do not apply to Medicare supplemental health insurance policies. However, Medicare supplemental health insurance policies are subject to similar genetic nondiscrimination requirements under section 104 of the Genetic Information Nondiscrimination Act of 2008 (Pub. L. 110-233), as incorporated into the NAIC Model Regulation relating to sections 1882(s)(2)(e) and (x) of the Act (The NAIC Model Regulation can be accessed at http://www.naic.org .). (6) Coverage supplemental to the coverage provided under Chapter 55, Title 10 of the United States Code (also known as CHAMPUS supplemental programs). (7) Similar supplemental coverage provided to coverage under a group health plan (as described in § 146.145(b)(5)(i)(C) of this subchapter). Response Yes / No Answer Criteria Yes: The plan is NOT a health plan and therefore not a covered entity. No: The plan is a health plan and therefore a covered entity. Logic DISABLEMENT LOGIC - Disabled if HIPAA3_9 is yes. Scoring Impact There is no scoring impact associated with this question.
Is the program one of the listed government health plans?	HIPAA4_1
Further Information Guidance/Context Response Yes / No Answer Criteria Yes: The plan is a health plan and therefore a covered entity. No: The plan is not a health plan and therefore not a covered entity Logic This is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Is the program an individual or group plan that provides, or pays the cost of, medical care?	HIPAA4_2
Further Information Guidance/Context An individual or group plan that provides, or pays the cost of, medical care. Health plans include government organizations (e.g., Medicaid, Medicare, and the Veterans Health Administration) Response Yes / No Answer Criteria Yes: The plan is a health plan and therefore a covered entity. No: The plan is NOT a health plan and therefore not a covered entity. Logic DISABLEMENT LOGIC - Disabled if HIPAA4_1 is yes. Scoring Impact There is no scoring impact associated with this question.
Is the program a high risk pool?	HIPAA4_3
Further Information Guidance/Context Similar to the Pre-Existing Condition Insurance Plan under the Affordable Care Act, for years many states have offered plans that provide coverage if you have been locked out of the individual insurance market because of a pre-existing condition. High-risk pool plans may also offer coverage if you're HIPAA eligible or meet other requirements. High-risk pool plans offer health insurance coverage that is subsidized by a state government. Typically, your premium is up to twice as much as you would pay for individual coverage if you were healthy. Response Yes / No Answer Criteria Yes: The plan is a health plan and therefore a covered entity. No: The plan is not a health plan and therefore not a covered entity. Logic DISABLEMENT LOGIC - Disabled if HIPAA4_2 is no. Scoring Impact There is no scoring impact associated with this question.
Is the plan a health maintenance organization (HMO)?	HIPAA4_4
Further Information Guidance/Context A type of health insurance plan that usually limits coverage to care from doctors who work for or contract with the HMO. It generally won't cover out-of-network care except in an emergency. An HMO may require you to live or work in its service area to be eligible for coverage. HMOs often provide integrated care and focus on prevention and wellness. Response Yes / No Answer Criteria Yes: The plan is a health plan and therefore a covered entity. No: The plan is not a health plan and therefore not a covered entity. Logic DISABLEMENT LOGIC - Disabled if HIPAA4_3 is yes. Scoring Impact There is no scoring impact associated with this question.
Is the principal activity of the program providing health care directly?	HIPAA4_5
Further Information Guidance/Context Direct healthcare is healthcare directly purchased by and delivered to an organization and its members, with no third party in between. Most often, the purchasing organization is a large, self-funded employer, or another aggregating entity like an association, trust, Taft-Hartley plan, or labor union. Response Yes / No Answer Criteria Yes: The plan is NOT a health plan and therefore not a covered entity. No: Logic DISABLEMENT LOGIC - Disabled if HIPAA4_4 is yes. Scoring Impact There is no scoring impact associated with this question.
Is the principal activity of the program the making of grants to fund the direct provision of healthcare (e.g. through funding a health clinic)?	HIPAA4_6
Further Information Guidance/Context Grant - a sum of money given by a government or other organization for a particular purpose, in this instance health care. Response Yes / No Answer Criteria Yes: The plan is NOT a health plan and therefore not a covered entity. No: Logic DISABLEMENT LOGIC - Disabled if HIPAA4_5 is yes. Scoring Impact There is no scoring impact associated with this question.
Is the principal purpose of the program other than providing or paying the cost of health care (e.g. operating a prison system, running a scholarship or fellowship program)?	HIPAA4_7
Further Information Guidance/Context Where the principal purpose of the program is not the provision or paying the cost of health care, then the plan is not a health plan and therefore the organization is not a covered entity. Response Yes / No Answer Criteria Yes: The plan is NOT a health plan and therefore not a covered entity. No: Logic DISABLEMENT LOGIC - Disabled if HIPAA4_6 is yes. Scoring Impact There is no scoring impact associated with this question.
Does the program provide only expected benefits?	HIPAA4_8
Further Information Guidance/Context The requirements of this part do not apply to any individual coverage in relation to its provision of the benefits described in paragraphs (a) and (b) of this section (or any combination of the benefits). (a) Benefits excepted in all circumstances. The following benefits are excepted in all circumstances: (1) Coverage only for accident (including accidental death and dismemberment) (2) Disability income insurance. (3) Liability insurance, including general liability insurance and automobile liability insurance. (4) Coverage issued as a supplement to liability insurance. (5) Workers' compensation or similar insurance. (6) Automobile medical payment insurance. (7) Credit-only insurance (for example, mortgage insurance). (8) Coverage for on-site medical clinics. (9) Travel insurance, within the meaning of § 144.103 of this subchapter. (b) Other excepted benefits. The requirements of this part do not apply to individual health insurance coverage described in paragraphs (b)(1) through (b)(6) of this section if the benefits are provided under a separate policy, certificate, or contract of insurance. These benefits include the following: (1) Limited scope dental or vision benefits. These benefits are dental or vision benefits that are limited in scope to a narrow range or type of benefits that are generally excluded from benefit packages that combine hospital, medical, and surgical benefits. (2) Long-term care benefits. These benefits are benefits that are either - (i) Subject to State long-term care insurance laws; (ii) For qualified long-term care insurance services, as defined in section 7702B(c)(1) of the Code, or provided under a qualified long-term care insurance contract, as defined in section 7702B(b) of the Code; or (iii) Based on cognitive impairment or a loss of functional capacity that is expected to be chronic. (3) Coverage only for a specified disease or illness (for example, cancer policies) if the policies meet the requirements of § 146.145(b)(4)(ii)(B) and (C) of this subchapter regarding noncoordination of benefits. (4) Hospital indemnity or other fixed indemnity insurance only if - (i) The benefits are provided only to individuals who attest, in their fixed indemnity insurance application, that they have other health coverage that is minimum essential coverage within the meaning of section 5000A(f) of the Internal Revenue Code, or that they are treated as having minimum essential coverage due to their status as a bona fide resident of any possession of the United States pursuant to Code section 5000A(f)(4)(B). (ii) There is no coordination between the provision of benefits and an exclusion of benefits under any other health coverage. (iii) The benefits are paid in a fixed dollar amount per period of hospitalization or illness and/or per service (for example, $100/day or $50/visit) regardless of the amount of expenses incurred and without regard to the amount of benefits provided with respect to the event or service under any other health coverage. (iv) A notice is displayed prominently in the application materials in at least 14 point type that has the following language: “THIS IS A SUPPLEMENT TO HEALTH INSURANCE AND IS NOT A SUBSTITUTE FOR MAJOR MEDICAL COVERAGE. LACK OF MAJOR MEDICAL COVERAGE (OR OTHER MINIMUM ESSENTIAL COVERAGE) MAY RESULT IN AN ADDITIONAL PAYMENT WITH YOUR TAXES.” (v) The requirement of paragraph (b)(4)(iv) of this section applies to all hospital or other fixed indemnity insurance policy years beginning on or after January 1, 2015, and the requirement of paragraph (b)(4)(i) of this section applies to hospital or other fixed indemnity insurance policies issued on or after January 1, 2015, and to hospital or other fixed indemnity policies issued before that date, upon their first renewal occurring on or after October 1, 2016. (5) Medicare supplemental health insurance (as defined under section 1882(g)(1) of the Social Security Act. 42 U.S.C. 1395ss, also known as Medigap or MedSupp insurance). The requirements of this part 148 (including genetic nondiscrimination requirements), do not apply to Medicare supplemental health insurance policies. However, Medicare supplemental health insurance policies are subject to similar genetic nondiscrimination requirements under section 104 of the Genetic Information Nondiscrimination Act of 2008 (Pub. L. 110-233), as incorporated into the NAIC Model Regulation relating to sections 1882(s)(2)(e) and (x) of the Act (The NAIC Model Regulation can be accessed at http://www.naic.org .). (6) Coverage supplemental to the coverage provided under Chapter 55, Title 10 of the United States Code (also known as CHAMPUS supplemental programs). (7) Similar supplemental coverage provided to coverage under a group health plan (as described in § 146.145(b)(5)(i)(C) of this subchapter). Yes: The plan is NOT a health plan and therefore not a covered entity.No: The plan is a health plan and therefore a covered entity. Response Yes / No Answer Criteria Yes: The plan is NOT a health plan and therefore not a covered entity. No: The plan is a health plan and therefore a covered entity. Logic DISABLEMENT LOGIC - Disabled if HIPAA4_7 is yes. Scoring Impact There is no scoring impact associated with this question.
Is there documentation or a statement on the site that indicates this app is a business associate to a covered entity, and therefore should be HIPAA compliant?	HIPPAA5_1
Further Information Guidance/Context A “business associate” is a person or entity that performs certain functions or activities that involve the use or disclosure of protected health information on behalf of, or provides services to, a covered entity. They have an obligation to comply with HIPAA in line with each covered entities own HIPAA compliance policies. Response Yes / No Answer Criteria Yes: The business or agency is a business associate and therefore is required to be HIPAA compliant. No: Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Is the organization required to be HIPAA compliant?	HIPAA6_3
Further Information Guidance/Context If the product has been identified as a covered identity or business associate then they are required to be HIPAA compliant. Response Yes / No Answer Criteria Yes: If any of the above responses indicate that the organization is a covered entity or business associate. No: If the above responses indicate that the organization is not a covered entity or a business associate. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Is there a statement or evidence that the organization is compliant with HIPAA requirements?	HIPAA6_1
Further Information Guidance/Context Covered entities would be expected to provide a statement to say they are compliant with HIPAA requirements. If they are a business associate then they would not be required to provide their own statement of HIPAA compliance but many of the rules and regulations would likely be found in their privacy statement e.g. minimum necessary rule, privacy rule etc. Response Yes / No Answer Criteria Yes: If there is a statement of compliance with HIPAA No: If there is no statement of compliance Logic DISABLEMENT LOGIC - Disabled if the developer has not been identified as a Covered Entity or Business Associate based on responses to HIPAA1_1 - HIPAA5_1. Scoring Impact Exceptionally high risk if the organization is required to be HIPAA complaint (HIPAA6_3 is yes) and there is no evidence of HIPAA compliance (HIPAA6_1 is no).
Is there a statement or evidence that the organization is HITRUST certified?	HIPAA6_2
Further Information Guidance/Context This question, for companies that are NOT covered entities, is a demonstration of being “HIPAA Ready”. For those that are covered entities, it is a way of demonstrating their compliance with HIPAA requirements. Response Yes / No Answer Criteria Yes: If the organisation displays a HITRUST badge or certificate. No: If there is no evidence of HITRUST certification. Logic There is no disablement logic written for this question. Scoring Impact High value if the organization is not required to be HIPAA compliant but is HITRUST certified Medium value if there is evidence the organization is HIPAA compliant (HIPAA6_1) and if the organization is HITRUST certified (HIPAA6_2).

Privacy Policy

Initially, the assessment identifies the relevant privacy policy for the app, which is available to users through the app and/or the App Store or Play Store. The more transparent the privacy policy, the better. Ultimately, the privacy policy must clearly state that user data will not be used or shared with other parties, except as described in the privacy policy, or without the express consent of the user. Ideally, it will identify:
· what data is collected from the user and how,
· if the user is informed of the developer’s intentions with processing and sharing their data, and
· if the user’s consent is obtained.
The privacy policy should accurately reflect the data usage of the app. The assessors will be able to note if any data is collected outside of what is detailed in the privacy policy. Additionally, the policy should inform users of the developer’s intent to use their data for marketing purposes. If user data is shared for any other purposes other than basic use of the app, or legal obligations, then the review considers if the user is able to opt-out of these activities.

Click here to expand...

Is there a Privacy Policy clearly available via the app?	ORC_D39a
Further Information Guidance/Context This question should only be active if personal and/or sensitive data is collected by the app and is shared with the developer/third parties etc. If this appears to be wrongly active then check back on data scene setters. Response Type Yes / No Answer Criteria Yes: If any data is collected by or through the app, in any way. Including data such as usage data, cookies etc. No: Logic DISABLED LOGIC -This question should only be active if personal and/or sensitive data is collected by the app and is shared with the developer/third parties etc. If this appears to be wrongly active then check back on data scene setters. Scoring Impact Maximum risk applied to the data section in this question and all questions that are disabled as a result of answering D39a as No. Questions would be disabled as a result of them not being applicable due to the app not having an applicable policy.
Is there a Privacy Summary published anywhere by the developer? (Only relevant to mobile pps)	ORC_D39b
Further Information Guidance/Context Due to the nature of the data being collected being non-identifiable a summary privacy is suitable. This question should only be active if personal and/or sensitive data is not collected by the app or is not shared with the developer/third parties etc. If this appears to be wrongly active then check back on data scene setters. Response Type Yes / No Answer Criteria Yes: A privacy summary can be a simple paragraph explaining privacy practices of the developer, as collection of non-personal/sensitive data does not require a full privacy policy. Logic DISABLED LOGIC -Disabled if data is automatically shared/collected. Only enabled when the app/developer collects only none sensitive data OR when the personal/sensitive data is only shared through direct manual intervention from the user. Scoring Impact Maximum risk applied to this question and all questions that are disabled as a result of answering D39b as No. Questions would be disabled as a result of them not being applicable due to the app not having an applicable policy.
Is the Privacy Policy made immediately available when the user first opens the app?	ORC_DP03
Further Information Guidance/Context This question looks to identify when a privacy policy is presented to a user. Making users aware of how their data is collected and used is mandatory. Response Type Yes / No Answer Criteria Yes: If the privacy policy is displayed when the app is first opened. Yes: If the user is prompted to view and/or provided with a link to the policy when the app is first opened or on the login page. Logic DISABLED LOGIC -Disabled if D39a AND D39b are answered no. Scoring Impact High value applied if Yes. Value cannot be applied for both DP03 and DP04.
Is the policy made available when the user is signing up to the service?	ORC_DP04
Further Information Guidance/Context This question looks to identify when a privacy policy is presented to a user. Making users aware of how their data is collected and used is mandatory. Response Type Yes / No Answer Criteria Yes: When the user is provided with the privacy policy during the sign up process. No: If the user is not provided with, or linked to the privacy policy during sign up. Logic DISABLED LOGIC - Disabled if D39a AND D39b are no, OR if DT14 is no. Scoring Impact High value applied if Yes. Value cannot be applied for both DP03 and DP04.
Is it published within the app?	ORC_DP01
Further Information Guidance/Context A privacy policy must be accessible to the user. This and following question look to identify where the privacy policy is located. Publishing within the app or being accessible via the app result in high value than it only being identifiable on the relevant app store. Response Type Yes / No Answer Criteria Yes: If the privacy policy is readily available to read at any time within the app. No: If the privacy policy link takes you out of the app to a web browser. Logic DISABLED LOGIC - Disabled if D39a AND D39b are answered no. Scoring Impact Medium value applied if Yes. Value cannot be applied for both DP01 and DP02 if both are answered Yes.
Is it available externally via the app, or via a linked website?	ORC_DP02
Further Information Guidance/Context A privacy policy must be accessible to the user. To determine if a policy is external, a user can enter the app manager screen. If still on the app the policy is internal, if an internet browser has opened separate from the the app then it is external. Response Type Yes / No Answer Criteria Yes: If the policy links outside of the app to the browser. Yes: If there is an external link to the website, where there is access the privacy policy. This comes under the 2 click rule. Meaning that a privacy policy is easily accessible within 2 clicks/taps. Logic DISABLED LOGIC - Disabled if D39a AND D39b are answered no. Scoring Impact Medium value applied if Yes. Value cannot be applied for both DP01 and DP02 if both are answered Yes.
Is it available via the relevant app store?	ORC_DP05
Further Information Guidance/Context A privacy policy must be accessible to the user. This and following question look to identify where the privacy policy is located. Publishing within the app or being accessible via the app result in high value than it only being identifiable on the relevant app store. Response Type Yes / No Answer Criteria Yes: If the policy is accessible through the app store, making sure the privacy policy applies to the app. If it doesn’t link directly make sure it is accessible within 2 clicks. Logic DISABLED LOGIC - Disabled if D39a AND D39b are answered no. Scoring Impact Low value applied if Yes.
Is the Privacy Policy placed in another prominent location that is easily accessible?	ATA_DP01
Further information Guidance/Context A privacy policy must be accessible to the user. This and following question look to identify whether a privacy policy has been made available to the user in a location different to those listed above. Response Yes/No Assessment Criteria Yes: Assessor is able to find a privacy policy elsewhere. No: The assessor is unable to locate a privacy policy in a alternate place to those listed above. Logic DISABLED LOGIC - Disabled if D39a AND D39b are answered no. Scoring Impact Low value applied if yes.
What data does the Privacy Policy state the developer collects?	ORC_DP06
Further Information Guidance/Context This is a multiple choice question. Choices should be selected based on what is stated in the privacy policy. This would normally be found in a section titled like “What Information do we collect?”. Choices should only be selected if the privacy policy states them, this question should not be based off what can be seen in the app. Response Type Multiple Choice Answer Criteria Sensitive - Physical / Mental Health or Condition (past, current or future status), Sexual Life / Orientation, Political, Religious or other beliefs or opinions, Offences Committed / Alleged to have Committed / Criminal Proceedings / Outcomes / Sentence, Financial data (that might be used for payment fraud), Trade Union membership, Racial / Ethnic Origin, Genetic or Biometric Data (e.g. fingerprints / facial Recognition) for the purpose of uniquely identifying a person Personal (combined - If a number of these items have been selected, then there is a possibility that data can be personally identifiable) - Cookies, web beacons, flash cookies, server logs etc which track individual’s browsing behaviour, Other Unique Device Identifiers eg. Device MAC Address, Name, Age/DOB, Gender (self declared or observed), Marital Status \|Family / Lifestyle / Social Circumstance, Education / Qualifications / Professional Training / Awards, Other online identifiers / Event Logs, Location Data (Travel / GPS / GSM Data / radio frequency identification tags (RFID)) Personal - Address\|Postcode (full), Email Address, Mobile Phone Number / Device Number / Home Phone Number, Physical Description, Username, IP Address, General Identifier e.g. NHS No, Income / Financial / Tax Situation, Employment / Career History, Device IMEI No Non-Personal - General Wellness data Logic DISABLED LOGIC - Disabled if D39a AND D39b are answered no. Scoring Impact There is no scoring impact for this question.
Is the policy accurate, with regards to the data the developer intends to collect?	ORC_DP07
Further Information Guidance/Context This questions looks to capture if the data that is stated as collected by the developer within the privacy policy matches what has been identified during assessment and usage of the app. Response Type Yes / No Answer Criteria Yes: If DP06 contains the same selections as DT10. Logic DISABLED LOGIC - Disabled if D39a AND D39b are answered no. Scoring Impact High risk applied if No AND DT10 and DP06 do not match.
Does the app explicitly state that data collected by the app is stored locally, unless the user manually exports the data?	ORC_D10a
Further Information Guidance/Context This question looks at whether data is stored within the app. An app that requires data to be automatically transferred off the app - even to just be stored remotely - would not meet this requirement. Response Type Yes/No Answer Criteria Yes: “Stored locally on device” is clearly stated. Yes: Data is stored only on the device, unless a user chooses to share it, or no data is collected or stored by the developer. No: Doesn’t state that personal data is stored only on the device. No: Personal data is clearly transferred to and stored in any location outside the device with no involvement from the user. Logic DISABLED LOGIC - Disabled if D01 OR DS06 are No, or if DS07 OR DS09 are answered Yes. Scoring Impact High value if Yes.
How does the developer obtain consent for the processing of user data?	ORC_DP08
Further Information Guidance/Context Consent should be obvious and require a clear, positive, physical action from the user to opt in. Consent requests must be prominent, separate from other terms and conditions, easy to understand, and user friendly. During sign-up to the app attention should be paid to how, if at all, consent is obtained from the user. Response Type Multiple Choice Answer Criteria Unmarked opt in check box, separate from other terms and conditions and/or consent requests (separate boxes for privacy policy, terms/conditions and marketing).- if there is an unmarked checkbox where the user can agree or consent to the privacy policy alone. Clear affirmative acceptance option, separate from acceptance of other terms and conditions and/or consent requests (separate acceptance option for privacy policy, terms/conditions and marketing).- if there is another form of acceptance of the privacy policy, eg. clicking “sign up” after having been presented with the privacy policy. Explicitly through express confirmation in words, rather than any other positive action (e.g. the user is required to email/write to the developer providing a clear confirmation of consent). This does not apply to a statement in the privacy policy such as “by using this app you consent to us collecting your data.”)- if the user is required to email the developer to provide their written consent. Another form of positive action to opt in to giving consent (please detail below) - eg. if the acceptance box is for both privacy policy and T&Cs.- if there is an unmarked checkbox to agree to the privacy policy and T&Cs all together. Other (please detail below), e.g. A statement in the privacy policy such as by using this app you consent to us collecting your data, with no clear confirmation of acceptance of policy. - if there is no clear option to be taken by the user to accept the privacy policy. Logic DISABLED LOGIC - Disabled if D39a is No. Scoring Impact Very high risk applied if “Other” is selected + multiplier based on nature of the data.
Does the Privacy Policy Provide the name and contact details of their Privacy Officer (PO), or similar individual representative for the company?	ORC_DP14
Further Information Guidance/Context A DPO is important to ensure, in an independent manner, that an organisation applies the laws protecting individuals' personal data. Response Type Yes/No Answer Criteria Yes: If an individual person has been named and declared the person responsible for the company’s privacy practices, with contact details (this can include a generic email, such as dpo@company.co.uk, providing the individual responsible has been named). No: If an individual person who is responsible for the role of DPO has not been named/detailed. No: If there is only a generic email address. Logic DISABLED LOGIC - Disabled if D39a is no. Scoring Impact High value applied if yes.
Provide the details of the representative: (Text response)	ORC_DP15
Further Information Guidance/Context Input the details of DPO from within privacy policy, this should be a named person not just a generic email and “data protection officer”. Response Type Free Text Answer Criteria N/A Logic DISABLED LOGIC - Disabled if DP14 has not been answered yes. Scoring Impact None

Data use

Once it is established what data is collected by the app, the assessment looks at how that data is used and shared, and if this is communicated to the user. The privacy policy should state all intended uses and legal basis of processing user data, such as legal obligation, research or marketing. Users should also be given the option to withdraw consent for the use of their data, particularly for marketing.

Click here to expand...

Does the developer fully inform the user of how they will collect data about them?	ORC_D69
Further Information Guidance/Context This questions identifies if the developers has clearly stated in the privacy policy how data will be collected from users. For example “ data will be collected when registering to use the app”. Response Type Yes/No Answer Criteria Yes: If the developer informs users where any data will be collected from. Eg. directly from the user or through third party sources. No: If the developer has not informed users of all potential sources of information about them. Eg. the user is informed of data collected about them, however, the developers fail to identify that information is obtained from another location, such as Facebook, when the user signs up with this account. Logic DISABLED LOGIC - Disabled if D39a and D39b have not been answered yes OR DS06 has been answered Yes. Scoring Impact Medium risk applied if No + multiplier based on the nature of the data.
Does the developer provide users with details on all the purposes of processing user data?	ORC_D13
Further Information Guidance/Context This question looks to identify if the purpose of processing has been made clear. For example, a developer may state that email addresses are captured to share marketing information with users. Response Type Yes/No Answer Criteria Yes: If the policy clearly explains what the user data collected is used for. Yes: If the policy states all the uses for collected data that are apparent from the app. No: If there is reason to believe that the developer has not explained any of the purposes for processing user data (Please detail in comments section). Logic DISABLED LOGIC - Disabled if D39a and D39b have not been answered yes OR DS06 has been answered yes. Scoring Impact Medium risk applied if No + multiplier based on the nature of the data.
What is automatically shared data used for?	ORC_DP10
Further Information Guidance/Context Selection of answers for this questions should apply to data automatically shared with third parties/HCP/other users/devices - NOT with the developer. The exception to this is marketing should be selected if this a purpose of data sharing with the developer. Response Type Multiple Options Answer Criteria Legal obligations, Performance of contract, Payment transactions, Research, Improving of developer services, Marketing, Provision of service, Other (Please specify), Unclear. Logic DISABLED LOGIC - Disabled if D39a and D39b have not been answered yes OR DS06 has been answered Yes. Scoring Impact None
Does the developer appear to intend to share or process the user data collected by the app for any purposes that have not been made clear to the user, or for any purposes they deem necessary?	ORC_D38
Further Information Guidance/Context This question is asking if there is the possibility that data is being shared without this being made clear to the user. Therefore, No is the positive response. Response Type Yes/No Answer Criteria Yes: If data is shared without user consent, AND users don’t need to agree to the privacy policy. Essentially the opposite of D16. Yes: If there is an obvious purpose for data use, which isn’t made clear or mentioned in the privacy policy. Yes: The policy states that the data will not be shared without first obtaining the user's consent to do so or that the app/developer ‘Won't share for other reasons/ with other parties, except as has been set out in the policy without obtaining your consent’ Logic DISABLED LOGIC - Disabled if D39a has not been answered Yes. Scoring Impact High risk applied if yes + multiplier based on the nature of the data.
Does the developer inform users that they would like to use their data for the purpose of marketing?	ORC_D71
Further Information Guidance/Context If direct marketing is being undertaken then developers need separate additional consent from the users. Answers for this questions are typically found in a section with name similar to “What we do with the information we collect” or the developers may have a separate “Marketing” section. Response Type Yes/No Answer Criteria N/A Logic DISABLED LOGIC - Disabled if D39a has not been answered yes or DT13 is answered as non-personal or D01 is answered no. Scoring Impact None
Does the developer obtain informed consent separately, for the purpose of marketing?	ORC_DP12
Further Information Guidance/Context Consent for marketing should be obtained separately from consent for any processing user data for any other purpose. This must also be prominent, easy to understand and user friendly. E.g. A separate tick box for marketing and consenting with the privacy policy. Response Type Yes/No Answer Criteria Yes: If consent for marketing is obtained separately for marketing AND the method for gaining this consent is through one of the positive affirmative actions listed in DP08. (Unmarked opt in check box; clear affirmative acceptance option; explicitly through express confirmation in words, another form of positive action to opt in to giving consent (please detail below)). No: If the user is not asked for consent to use data for marketing separately. No: If the user has not been required to provide a positive affirmative action, separate from accepting other T’s & C’s / Privacy Policies, to agree to sharing their data for the purposes of marketing. Logic DISABLED LOGIC - Disabled if DP10 does not contain Marketing, or if DT13 is answered as non-personal. Scoring Impact High risk applied if no + multiplier based on the nature of the data.
Does the developer obtain informed consent separately, for the purpose of marketing, as defined under HIPAA?	ORC_ATA_DP02
Further Information Guidance/Context The HIPAA Privacy Rule gives individuals important controls over whether and how their protected health information is used and disclosed for marketing purposes. With limited exceptions, the Rule requires an individual’s written authorization before a use or disclosure of his or her protected health information can be made for marketing. So as not to interfere with core health care functions, the Rule distinguishes marketing communications from those communications about goods and services that are essential for quality health care. Response Type Yes/No Answer Criteria Yes: If a separate consent is gained for communicating about goods and services that are essential for quality health care. No: If a separate consent is not gained for marketing as defined under the HIPAA privacy rule. Logic DISABLED LOGIC - Disabled if D01 is no, DS06 is yes, D39a is no OR D71 is no. Scoring Impact Medium - high risk applied if no.
Is the user informed of how they can opt out of each of these activities?	ORC_D28
Further Information Guidance/Context The list of activities can be found in question ORC_DP10. The developer should state how a user can opt out of each of these processing activities. Response Type Yes/No Answer Criteria Yes: If the app has an option to opt out/turn off data collection for external research or provides a contact email to get data removed from a study. Yes: If the policy clearly explains to user how they can contact the developer to opt-out of all sharing/processing activities. No: If shared for any other reasons other than legal obligations and no option to opt out (email address in policy explicitly stating how to opt out or sliders within app). No: If the policy only mentions how the user can opt out of one, but not all, activities. Logic DISABLED LOGIC - Disabled if D39a and D39b have not been answered yes. Scoring Impact Medium risk applied + multiplier based on the nature of the data. Risk can not be applied to both D28 and DP13.
If the user can not opt out of all processing activity, does the developer clearly explain which activities they cannot opt out of and why?	ORC_DP13
Further Information Guidance/Context N/A Response Type Yes/No Answer Criteria Yes: If only shared for legal obligations - Policy must state who they will share the data with and for what legal purposes (e.g. protect rights, copyright). Yes: If the developer has clearly set out justifiable reasons for not being able to deal with particular requests with regards to stopping certain processing/sharing activities. No: If users are not informed of how they can either opt out of processing and sharing activities AND there is no justification from the developer as to why users cannot opt out of certain activities. Logic DISABLED LOGIC - Disabled if D39a and D39b have not been answered yes or D28 has been answered Yes. Scoring Impact Medium risk applied if no + multiplier based on the nature of the data. Risk can not be applied to both D28 and DP13.
Is the user informed that their data will not be shared with other parties, except for the purposes that have been set out in the privacy policy?	ORC_D16
Further Information Guidance/Context Developers are required to share who data may be shared with for processing and other activities. Information for this question will typically be located in the privacy policy around information about what third parties data is shared with. Response Type Yes/No Answer Criteria Yes: If no data is shared without user consent. Yes: If the policy states that using the app indicates agreement to the policy/given consent for data sharing specified. No: If data is shared with third parties without user consent. Logic DISABLED LOGIC - Disabled if D39a has not been answered yes or D01 has been answered No or if DT13 has been answered non-personal. Scoring Impact High risk applied if no + multiplier based on the nature of the data.

Data Storage and Transit/Transfer

The key areas in this section are surrounding data storage and data transfer. The data privacy policy should inform the user of where their data is stored, how their data is protected in storage, and how it is protected in transit between the user’s device and the host storage. The DHAF looks for specific and secure storage techniques, such as industry-recognized encryption or firewalls. During transit, it is preferable that data is protected using industry-recognized encryption. A list of deprecated encryption methods will be provided to assessors and regularly updated to ensure the assessment is kept up-to-date with current industry practices.

Click here to expand...

Does the data privacy policy or equivalent provide detail about where the data collected by the app will be stored (i.e. on the app or in an external data warehouse, cloud server etc.)?	ORC_DST01
Further Information Guidance/Context N/A Response Type Yes/No Answer Criteria Yes: If the policy states the data is stored in a cloud server e.g. physical location e.g. secure server. Microsoft Azure is a cloud storage technology. AWS - amazon web servers. If policy states the physical address of the data controller. “May not be stored in your location” isn’t enough. “In the UK” isn’t enough, has to be an address. Logic DISABLED LOGIC - Disabled if D39a And D39b have not been answered yes. Scoring Impact Medium value applied if yes.
Where is the data stored?	ORC_DST02
Further Information Guidance/Context The purpose of this question is to state where data is stored e.g. in a secure server or an AWS server. Response Type Free Text Answer Criteria N/A Logic DISABLED LOGIC - Disabled if DST01 is no. Scoring Impact There is no scoring impact for this question.
Does the data privacy policy, or equivalent, state whether personal data is stored using recognised secure data storage technologies?	ORC_DST03
Further Information Guidance/Context This question is looking to identify if appropriate technologies are being used for secure storage of user data. Technologies that are considered appropriate are detailed in the answer criteria below. Response Type Yes/No Answer Criteria Yes: If firewall, antivirus, or encryption when in storage/at rest is mentioned. Must state which technology is used, this does not have to be specific. If AWS, Microsoft Azure or Google cloud server is mentioned. “256-bit SSL encryption for data transfer and NSA-level 256-bit AES encryption of personal information” - would be “yes” to both encryption in storage and transit. AES is yes to storage and encryption. LUKS (Linux Unified Key Setup) is yes. No: Doesn’t state which technology is used. Only mentions “cloud services” but doesn’t specify provider. Logic DISABLED LOGIC - Disabled if D39a And D39b have not been answered yes. Scoring Impact High value if yes. High risk if no.
Is all personally identifiable data encrypted in transit between the device and any external host storage?	ORC_D17
Further Information Guidance/Context The purpose of this question is to ensure that data is transferred securely to ensure there are no breaches of users data. Response Type Yes/No Answer Criteria Yes: If the policy states encrypted during transit or mentions the encryption type. Encryption types: If SSL or TLS = encryption in transit. HTTPS Web apps - check the address bar, the padlock means HTTPS. “256-bit SSL encryption for data transfer and NSA-level 256-bit AES encryption of personal information” - would be “yes” to both encryption in storage and transit. AES (Yes to both storage and encryption). No: Doesn’t state the data is encrypted during transit. Only mentions payment details are encrypted in transit. Logic DISABLED LOGIC - Disabled if D39a has not been answered yes. Scoring Impact High risk applied if no.
Is the user informed that online video consultations use secure encryption methods?	ORC_DST04
Further Information Guidance/Context Developers need to state that video consultations use secure encryption clearly if appropriate. This should be in addition to the previous question about encryption of other data transfer. Response Type Yes/No Answer Criteria Yes: If it is made explicitly clear to the end-user, that a secure encrypted connection is used for all video consultations. This may be in the policy or elsewhere on the website/app. Logic DISABLED LOGIC - Disabled if D39a has not been answered yes. Scoring Impact High risk applied if no.

Data Standards and Management

The DHAF will award additional points if an app developer is compliant with any recognized US Data Management Standards such as Software Development Life Cycle (SDLC), or any International Data Management Standards such as ISO 27001. The privacy policy should inform users of a data retention period, and a method for data destruction. The DHAF also identifies whether the developer has a policy in place to deal with any data security breaches.

Click here to expand...

Does the policy state its compliance with recognized Data Management Standards?	ORC_ATA_DP03
Further Information Guidance/Context Developers that are compliant with these international data standards are rewarded for compliance with best practice standards of data management. Response Type Multiple Option: ISOC2 ISO 27001 SDLC NIST Other (please specify). Answer Criteria Yes: If there is a compliance sticker on their website. ISO 27001. (if any other ISOs/BSIs etc. are mentioned, please confirm the appropriateness of the standard for data management. Yes: Needs to be the COMPANY that is ISO compliant, not the server where data is stored, particularly when the company/developer is the data controller. No: If there is no evidence of ISO, BSI etc. compliance. EU-US privacy shield does not count for this question. If the server (e.g. AWS) is ISO compliant but there is no explicit statement to say the company is. Logic DISABLED LOGIC - Disabled if D39a and D39b have not been answered yes. Scoring Impact Medium risk applied if no + multiplier based on the nature of the data.
Does the policy contain details of the length of time data is retained?	ORC_D19
Further Information Guidance/Context Under GDPR, it is a legal obligation for a data controller and/or processor to only retain personal data only for as long as is necessary for the purpose upon which it is being processed. Response Type Yes/No Answer Criteria Yes: If the developer mentions any time period of data retention, even if it’s an indefinite amount of time. Yes: If the developer states “We only keep your personal information for as long as it’s necessary for our original legitimate purpose for collecting it and for as long as we have your permission to keep it.” No: If the only mention of data retention is provided where the developer informs users of the rights under GDPR. No: Developers are obliged to separately inform users of their own policies and procedures regarding data retention in the event that a user has not exercised any of their rights to their data. No: If the policy mentions a timeframe after which data may be stored in aggregate. No: If the policy states “we may retain data for…” Logic DISABLED LOGIC - Disabled if D39a has not been answered yes. Scoring Impact Low risk applied if no.
Is there a statement containing details of a method for data destruction?	ORC_D20
Further Information Guidance/Context Controllers and/or processors should ensure there are set procedures in place to safely and securely delete any personal data when it is no longer needed and. Response Type Yes/No Answer Criteria Yes: If the policy mentions how the data is deleted, if users haven’t exercised any user rights. Yes: If the policy details that user data is deleted after a certain time period. Yes: If users can delete or reset all data within the app, AND it deletes it from the server. Not if it clears the app but stays on the server. Yes: If users are the data controller and the method of deletion is users contacting the developer to remove their data. Yes: If the developer has detailed the process for anonymizing personally identifiable information after a given timeframe of inactivity. No: If the only mention for deletion of data is provided where the developer informs users of the rights under GDPR. Developers are obliged to separately inform users of their own policies and procedures regarding the deletion of data if a user has not exercised any of their rights to their data. No: If the only mention is removal of data for under 13s/minors. Logic DISABLED LOGIC - Disabled if D39a has not been answered Yes. Scoring Impact Low risk score applied if D20 has been answered No, + multiplier based on the nature of the data collected.
Is there a statement that sets out a process for managing data confidentiality breaches?	ORC_D21
Further Information Guidance/Context Developers have an obligation to notify the relevant supervisory authority when certain data breaches happen. Developers should therefore have a clear internal procedure in place to help aid the decision-making about whether or not a breach needs to be reported to the supervisory authority or even the affected individuals. Response Type Yes/No Answer Criteria Yes: If users are informed that they can complain to the Information Commissioner's Office (ICO), if they believe that their data privacy rights have been breached. Yes: If users are informed they can lodge a complaint with their local data protection authority, if they believe that their data privacy rights have been breached. Yes: If users are informed that they can complain to the Local Supervisory Authority of the country that the developer is based, if they believe that their data privacy rights have been breached. Yes: If the user is told that they should inform the company, or the company will inform the user, (if you suspect a breach) and users have the right to file a complaint with the competent supervisory authority (GDPR Art. 77). Check T&Cs too. Yes: If the developer has detailed in the privacy policy how they will approach any breaches to data security that they become aware of. For example informing users within a reasonable time frame and informing their relevant jurisdictional supervisory authority. Yes: If the developer has detailed the process in which they will inform the local/jurisdictional regulatory authority of any confidentiality breaches. No: If the policy doesn’t state what happens in the event of a breach. No: If you can complain but only to the developer, not to the ICO in the event of a breach. Logic DISABLED LOGIC - Disabled if D39a has not been answered Yes. Scoring Impact Medium risk applied if D21 has been answered No + multiplier based on the nature of the data collected.

GDPR/HIPAA/Jurisdictional Principles

This review area focuses on Data Protection based on UK and EU General Data Protection Regulation (GDPR). The DHAF is concerned that all apps, particularly those developed in the UK and the EU, are fully compliant with the GDPR and is applied here as best practice in the assessment. This means a clear and explicit statement of compliance, as well as confirming that the user is entitled to the 7 user rights.
The developer should also inform the user of how they can exercise these rights and should commit to responding within a time frame of 2 months or less. Under the GDPR, the policy should outline the legal basis for the collection of user data, and ensure that only minimal data is collected from the user.
Similar and additional requirements and user rights are observed under HIPAA and have been built into this section to be specifically adapted to the DHAF.
All questions relating to this section will only be asked for apps that collect and process personal and/or sensitive data, as a measure of either HIPAA Compliance or “HIPAA Readiness”.

Click here to expand...

Is there a statement that confirms the app’s compliance with jurisdictionally required laws and regulations?	ATA_DP05
Further information Guidance/Context Jurisdictionally required principle recognition, would be that the developer recognizes their responsibility, at a minimum, to treat the data of an individual from another country in compliance with the data protection laws of the country in which that individual resides. Response Yes / No Assessment Criteria Yes: The assessor finds a statement confirming the app’s compliance with jurisdictionally required laws and regulations. No: There’s no statement confirming the app’s compliance with jurisdictionally required laws and regulations. Logic DISABLED LOGIC - Disabled if D39a and D39b are no. Scoring Impact Medium risk applied if no + multiplier based no nature of the data collected.
Is the user informed of the legal basis for which data is collected from them?	ORC_D60
Further Information Guidance/Context To meet the requirement for this question the developer has to specify ‘the legal basis for data collection is…’ Response Type Yes/No Answer Criteria Yes: If the policy states data is collected under a legal basis, e.g. consent, performance of contract, legal obligation, vital interests, public interest or legitimate interest. Yes: If the policy states that by using the app you consent to the privacy policy, “If you consent to this app, you consent for us to collect data.”, or a statement similar to this. Logic DISABLED LOGIC - Disabled if D39a has not been answered Yes. Scoring Impact High risk score applied if D60 has been answered No + multiplier based on the nature of the data collected.
Is the user informed of the developer’s intent to ensure that only the “Minimum Necessary” data /PHI, as defined under HIPAA are collected?	ATA_DP06
Further information Guidance/Context https://www.hhs.gov/hipaa/for-professionals/privacy/guidance/minimum-necessary-requirement/index.html Response Yes / No Answer Criteria N/A Logic DISABLED LOGIC - Disabled if D39a has not been answered Yes. Scoring Impact High - Exception risk applied if no + multiplier based on if the organization is required to be HIPAA compliant and based on the nature of the data the app collects.
Is there a statement that the policy will be updated duly should the purpose of data collection change? This may mean re-obtaining consent (if consent was the lawful basis).	ORC_D61
Further Information Guidance/Context The developer has an obligation to inform users of any changes that are made to the processing of data. The level to which the developer must notify is determined by the legal basis for processing and the extent of the change being made. Response Type Yes/No Answer Criteria Yes: If the legal basis is consent and the developer states that if the purpose for processing data changes then consent will be re-obtained before continued use of the service. Yes: If consent is not one of the legal basis and the developer has stated in the privacy policy that they WILL inform users of changes to the policy. No: If the developer states that they MAY inform the users of changes to the privacy policy. Logic DISABLED LOGIC - Disabled if D39a has not been answered Yes. Scoring Impact Medium risk score applied if D61 has been answered No + multiplier based on the nature of the data collected.
Are users informed of their rights with regards to their data?	ORC_DPR01
Further Information Guidance/Context Questions relating to GDPR will only be asked for apps that collect and process personal and/or sensitive data. Response Type Yes/No Answer Criteria Yes: If the developer has made it clear that the user has certain rights with regards to their data and explains what those rights are. Yes: If the developer has set out any of the user rights under GDPR. Logic DISABLED LOGIC - Disabled if D39a has not been answered Yes. Scoring Impact Exceptional risk score applied if DPR01 has been answered No + multiplier based on the nature of the data collected.
Has the developer made the existence of the data subject’s right to request that their personal data is deleted clear?	ORC_D93
Further Information Guidance/Context This is a right users should expect under GDPR. Response Type Yes/No Answer Criteria Yes If the policy clearly states the user’s right to erasure, or method for how data is deleted, OR the user can clear all data from the app. Yes: If the policy clearly states the user’s right to be forgotten. Logic DISABLED LOGIC - Disabled if D39a Or DPR01 have not been answered Yes. Scoring Impact Low value score applied if D93 has been answered Yes.
Has the developer made the existence of the data subject’s right to access their personal data clear?	ORC_D25
Further Information Guidance/Context This is a right users should expect under GDPR. Response Type Yes/No Answer Criteria Yes: If the policy clearly states the user’s right to access their data, and a contact method is given. Logic DISABLED LOGIC - Disabled if D39a Or DPR01 have not been answered Yes. Scoring Impact Low value score applied if D25 has been answered Yes.
Has the developer made the existence of the data subject’s right to inspect their personal data clear?	ATA_DP07
Further information Guidance/Context This statement should be clearly displayed as HIPAA requires data controllers to allow the user the right to inspect their personal data. Response Yes / No Assessment Criteria Yes: If the assessor can find the user right clearly displayed in the privacy policy or in another accessible location to the user. No: If the assessor is unable to find this user right. Logic DISABLED LOGIC - Disabled if D39a have not been answered Yes. Scoring Impact Low value applied if yes. Low risk applied if no.
Is the user informed of their rights to know how their PHI is used and or shared?	ATA_DP08
Further information Guidance/Context This statement should be clearly displayed as HIPAA requires data controllers to allow the user the right to be informed of how their PHI is used and / or shared. Response Yes / No Assessment Criteria Yes: If the assessor can find the user right clearly displayed in the privacy policy or in another accessible location to the user. No: If the assessor is unable to find this user right. Logic DISABLED LOGIC - Disabled if D39a have not been answered Yes. Scoring Impact Low value applied if yes. Low risk applied if no.
Has the developer made the existence of the data subject’s rights to rectify their personal data clear?	ORC_D56
Further Information Guidance/Context This is a right users should expect under GDPR. Response Type Yes/No Answer Criteria Yes: The policy clearly states the user’s right to rectify, correct, amend or update their information. Logic DISABLED LOGIC - Disabled if D39a Or DPR01 have not been answered Yes. Scoring Impact Low value score applied if D56 has been answered Yes.
Has the developer made the existence of the data subject’s rights to restrict the use of their personal data clear?	ORC_D81
Further Information Guidance/Context This is a right users should expect under GDPR. Response Type Yes/No Answer Criteria Yes: If the policy clearly states the user’s right to restrict use, or to stop using data. Logic DISABLED LOGIC - Disabled if D39a Or DPR01 have not been answered Yes. Scoring Impact Low value score applied if D81 has been answered Yes.
Has the developer made the existence of the data subject’s rights to object to the processing of their personal data clear?	ORC_D57
Further Information Guidance/Context This is a right users should expect under GDPR. Response Type Yes/No Answer Criteria Yes: If the policy clearly states the user’s right to restrict use, or to stop using data. Logic DISABLED LOGIC - Disabled if D39a Or DPR01 have not been answered Yes. Scoring Impact Low value score applied if D81 has been answered Yes.
Has the developer made the existence of the data subject’s rights to portability of (receive) their personal data clear?	ORC_D59
Further Information Guidance/Context This is a right users should expect under GDPR. Response Type Yes/No Answer Criteria Yes: If the policy clearly states the user’s right to portability, or right to transfer their data, or the right to receive their data in a machine-readable format. Logic DISABLED LOGIC - Disabled if D39a Or DPR01 have not been answered Yes. Scoring Impact Low value score applied if D59 has been answered Yes.
Has the developer made the existence of the data subject’s right to withdraw consent for the use of their personal data clear?	ORC_D58
Further Information Guidance/Context This is a right users should expect under GDPR. Response Type Yes/No Answer Criteria Yes: If the policy clearly states the user’s right to withdraw consent. Logic DISABLED LOGIC - Disabled if D39a Or DPR01 have not been answered Yes. Scoring Impact Low value score applied if D58 has been answered Yes.
Has the developer explained individual rights to users in a manner that is easily understood?	ATA_DP09
Further information Guidance/Context HIPAA requirement. In the policy, the developer should be clear that the user has this right, but also be clear of the repercussions of refusing to share data (eg. being unable to access/provide the service). Response Yes / No Assessment Criteria Yes: If the user rights have been made clear in the privacy policy and the repercussions of refusing to share data has also been made clear. No: If only the user rights have been made clear or if the repercussions have not been made clear. No: If the privacy policy has not mentioned user rights or repercussions of not consenting to sharing data. Logic DISABLED LOGIC - Disabled if D39a has not been answered Yes. Scoring Impact Medium- High risk applied if no + multiplier based on the nature of the data collected.
Has the developer been clear as to the repercussions of refusing to share or allow the processing of data?	ATA_DP10
Further information Guidance/Context In the policy, the developer should be clear that the user has this right, but also be clear of the repercussions of refusing to share data (eg. being unable to access/provide the service). Response Yes / No Assessment Criteria Yes: If repercussions of refusing to share data has also been made clear. No: If the privacy policy has not mentioned the repercussions of not consenting to sharing data. Logic DISABLED LOGIC - Disabled if D39a has not been answered Yes. Scoring Impact Medium- High risk applied if no + multiplier based on the nature of the data collected.
Has the developer made clear the existence of the user’s right to request that they are not subject to a decision based solely on automated processing, including profiling, which produces legal/significant effects concerning the user?	ORC_DPR02
Further Information Guidance/Context Automated processing is what can occur when applying for things such as insurance, finance, mortgage etc. It gives an output which is based on details entered. The result would be a machine driven decision or figure of cost etc. Users have the right to request any such decision be reviewed by a human. Response Type Yes/No Answer Criteria Yes: There may be a simpler statement, such as “You have the right to request that we do not process your personal data for the purpose of automated decision making”. Yes: If the developer has made clear that the user has this right under GDPR, even if they do not specifically process data in such a way. No: If this user right has not been mentioned in the policy. Logic DISABLED LOGIC - Disabled if D39a Or DPR01 have not been answered Yes. Scoring Impact Low value score applied if DPR02 has been answered Yes.
Does the developer provide details which the user can contact them on to exercise their rights?	ORC_D82
Further Information Guidance/Context When informing users of their individual rights under GDPR, it is also best practice to provide them with details on how the controller can be contacted/communicated with in order to submit subject access requests. Response Type Yes/No Answer Criteria Yes: If a contact method is provided in the policy for the developer, in relation to exercising user rights. No: If a contact method is only provided for one user right, rather than all rights mentioned. Logic DISABLED LOGIC - Disabled if D39a Or DPR01 have not been answered Yes. Scoring Impact Low value score applied if D81 has been answered Yes.
Is the user informed of the time frame in which the developer will respond to any requests to exercise their rights?	ORC_D83
Further Information Guidance/Context By law an organization normally has to respond to a subject access request within one month. If an individual has made a number of requests or a request is complex, extra time may be needed to consider and/or action the request(s). Where this is the case, the organization can take up to an extra two months to respond. Response Type Yes/No Answer Criteria Yes: If a time frame is given, and it is within two months of receipt of the request. No: If there is no separately provided timeframe and response commitment provided with regards to the user exercising their rights. I.e. if there are only contact details for enquiries about the policy as a whole, with an expected response time. Logic DISABLED LOGIC - Disabled if D39a Or DPR01 have not been answered Yes. Scoring Impact Low value score applied if D81 has been answered Yes.
Is the user informed of any charges that might be incurred with regards to exercising their rights to access their PHI?	ATA_DP11
Further information Guidance/Context It is a HIPAA requirement for the user to be informed of any charges which may be incurred with regards to exercise their right to access their PHI. Response Yes / No Assessment Criteria Yes: If users have been informed of charges that might be incurred when exercising their rights No: If users have not been informed that charges may be incurred when exercising their rights. Logic DISABLED LOGIC - Disabled if D39a has not been answered Yes. Scoring Impact Medium value applied if the organization does not need to be HIPAA compliant and this is answered yes. High Risk applied if the organization needs to be HIPAA compliant and this is answered no.
Is the user informed of their right to have an access denial reviewed?	ATA_DP12
Further information Guidance/Context It is a HIPAA requirement for the user to be informed of their right to have an access denial reviewed. Response Yes / No Assessment Criteria Yes: If users are informed that they have the right to request an access denial is reviewed. No: If users are not informed of their right to have an access denial reviewed. Logic DISABLED LOGIC - Disabled if D39a has not been answered Yes. Scoring Impact Medium value applied if the organization does not need to be HIPAA compliant and this is answered yes. High Risk applied if the organization needs to be HIPAA compliant and this is answered no.
Is the user informed that local state laws, providing them with additional rights with regards to their data, are not pre-empted by HIPAA?	ATA_DP13
Further information Guidance/Context It is a general rule under HIPAA that local state laws are not pre-empted by HIPAA. It is also best practice to ensure users are informed of this. Response Yes / No Assessment Criteria Yes: If users are informed that local state laws, are not pre-empted by HIPAA. Logic DISABLED LOGIC - Disabled if D39a has not been answered Yes. Scoring Impact Medium - High Risk applied if no + multiplier based on nature of the data.
Is the user informed of their right to request to be reached somewhere other than home?	ATA_DP14
Further information Guidance/Context It is a HIPAA requirement for the user to be informed of their right to request to be reached somewhere other than home. Response Yes / No Assessment Criteria Yes: If policy states that the user has the right to be reached somewhere other than home. No: If the policy has not mentioned this right. Logic DISABLED LOGIC - Disabled if D39a has not been answered Yes. Scoring Impact Medium value if yes and the organization is not required to be HIPAA compliant. High risk applied is no and the organization is required to be HIPAA compliant.

CLINICAL ASSURANCE & SAFETY

Professional Assurance

Validating the safety and efficacy of a Digital Health Technology (“DHT”) is a key part of any assurance process.

The Evidence Standards for Digital Health Technologies Framework (“ESF”) was created by the UK’s National Institute for Health and Care Excellence (“NICE”). This framework clustered DHT’s into relevant Tiers and identified for each Tier what forms of ‘evidence’ or ‘assurance’ would be required. It is therefore better to think of the ESF as an Assurance Standards Framework, with evidence being just one of many elements within that digital assurance matrix.

An adapted version of the ESF has been developed over time with and has now been adopted in numerous other national and pan-national Digital Health Assessment Frameworks in areas like the NORDICS, New Zealand, Canada, Israel and the Netherlands.

In addition the Clinical Evidence section looks for research backed evidence to support behavioral change techniques within the app and that the development of the app has involved suitably qualified professionals or validated with recognized organizations or bodies.

Evidence of Effectiveness

This is examined using an Evidence Standards Framework. We conduct an analysis of any evidence available through the Review Resources. If this exists, the app is evaluated against a series of questions to determine the quality of this evidence. We look for:
· a suitable sample size and makeup;
· a p-value of below 0.05 to indicate significance;
· a p-value below 0.2 for near significance; and
· an appropriate comparator.
This is scaled against the NICE Evidence Standards Framework and we look for a higher level of evidence for apps with more complex functionality and higher risk.

Question Set

What type/s of research article/study about the app is available? Survey, RCT, Pilot study, Observational (Case study, Cross-sectional, Cohort), Meta-Analysis/Systematic Review	ORC_EE02
Further Information Guidance/Context The purpose of this question is to identify the evidence that is available. Varying levels of evidence are required to pass the designated ESF tier. Choose all applicable from: survey, RCT, pilot study, observational study (including case study, cross-sectional or cohort), meta-analysis/systematic review, or indicated user acceptance/benefit. The follow on questions will be answerable for each evidence type chosen. Response Type Multiple Choice Answer Criteria Survey: If the app has gotten information from current users on their outcomes or how they utilize the app, and provides a description of the outcomes. Randomized Control Trial: The research paper will state this. An RCT has two (or more) groups of people, where the only major difference should be the treatment they receive, and as the name suggests, people should be randomly assigned to these groups. Pilot Study: A smaller-scale, preliminary study which is completed first to determine whether a study is feasible. A Case study with no p value would come under Pilot. Observational Study: An experimental or quasi-experimental study which demonstrates relevant outcomes. For example, a cohort study of individuals using a depression app. They measure depression before first use, and depression after eight weeks, and compare to see if there is any effect. This type of study also includes cross-sectional studies, which provide an image of people at a certain point in time. For example, it may be that people suffering pre-diabetes use an app. At the point in time studied, they have not developed type 2 diabetes, so the app may have helped. A Case Study would come under Observational Study, if it has a p value A cross-sectional study would come under Observational Study Meta-analysis/Systematic Review: A systematic review refers to the entire process of selecting, evaluating, and collating all available evidence, while the term meta-analysis refers to the statistical approach to combining the data derived from a systematic-review. For our review, this may be that the evidence provided has pulled together all the studies about an app to provide a single p-value to demonstrate the app’s significance. Indicated user acceptance/benefit: A statement or other piece of information which indicates a benefit of the app to users, or indication that the app has undergone a pilot study. This option is to be selected when you are unable to see further evidence that supports any claimed facts or outcomes. For example, the developer website states “9/10 users found their sleep improved”, but you can’t see the evidence behind this statement. Testimonials on the website can be accepted, but not from the app store review section. Any statement of users benefiting from the app. None: The assessor could not find any evidence supporting the app’s efficacy or functionality. Logic There is no disablement logic written for this question. Scoring Impact Varying amounts of risk or value applied depending on the defined ESF tier of the app. Higher tiered apps require more substantial evidence (i.e. an RCT study), if this is not identifiable and the app fails other criteria to pass at it’s defined ESF tier then more risk is applied. If the evidence is appropriate to the designated ESF tier and the app meets other criteria to pass it’s ESF tier then value is applied.
How many pieces of evidence does the app provide?	ORC_EE14
Further Information Guidance/Context The assessor is to select the option depending on how many pieces of evidence they have found. For each piece of evidence up to 5, you have to complete the following questions. Anymore than 5, you should choose the 5 best pieces. If multiple user acceptance statements have been found on one webpage, this would count as 1 piece of evidence. Response Type Multiple Option Answer Criteria One Two Three Four Five More than Five Logic DISABLEMENT LOGIC - Disabled if EE02 contains none. Scoring Impact There is no scoring impact associated with this question.
How many RCT's and/or observational studies does the app have?	ORC_EE13
Further Information Guidance/Context The assessor is to select the number of RCTs and observational studies they found which supports the app. For each piece of evidence up to 5, you have to complete the following questions. Anymore than 5, you should choose the 5 best pieces. Response Type Multiple Option Answer Criteria One Two Three Four Five More than Five Logic DISABLEMENT LOGIC - Disabled if EE02 does not contain RCT OR Observational. Scoring Impact There is no scoring impact associated with this question.
For each type of relevant research article: What category does the evidence relate to?	ORC_EE10
Further Information Guidance/Context This is about the ORCHA category to which the app relates. It may be there is evidence for more than one category, if this is the case input all that apply. Response Type Yes/No Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if EE02 contains none. Scoring Impact There is no scoring impact associated with this question.
For each type of relevant research article: What benefit does the evidence relate to?	ORC_EE11
Further Information Guidance/Context This is about the ORCHA benefit to which the app relates. It may be there is evidence for more than one benefit. Response Type Yes/No Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if EE02 contains none. Scoring Impact There is no scoring impact associated with this question.
Provide links to the publicly available evidence/published evidence that the developer has provided.	ORC_EE03
Further Information Guidance/Context This is a free text option, the answer should contain the links to the evidence found. Only place one link in the text box for each of these. Response Type Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if EE02 contains none. Scoring Impact There is no scoring impact associated with this question.
For each type of relevant research article: Is the sample size appropriate? (Does each sample /group contain 30?)	ORC_EE04
Further Information Guidance/Context This is about the sample size of any type of RCT or observational study that has been identified. There are no scoring implications as this is for data collection only. Response Type Yes/No Answer Criteria Yes: If the sample size is equal to or above 30 participants. No: If the sample size is below 30 participants OR the sample size is not mentioned. Logic DISABLEMENT LOGIC - Disabled if EE02 contains none. OR disabled if EE02 does not contain RCT or Observational. Scoring Impact There is no scoring impact associated with this question.
For each type of relevant research article: Does the research article found provide a p-value?	ORC_EE05
Further Information Guidance/Context This question aims to evaluate the significance demonstrated within the findings of the research provided above. Confidence intervals are also accepted along with p values as they too can demonstrate significance. Response Type Yes/No Answer Criteria Yes: If the research paper/article provides a p-value/confidence interval (CI). This will likely be found within the abstract and/or results section. Confidence interval example - (95% CI: 6.4–7.2). No: If the research paper/article does not provide a p-value. Logic DISABLEMENT LOGIC - Disabled if EE02 does not contain RCT or Observational. Scoring Impact There is no scoring impact associated with this question.
For each type of relevant research article: Does the P-Value demonstrate significance (p<0.05)?	ORC_EE06
Further Information Guidance/Context This question is used to provide indication on whether or not the research article has proven a benefit. There are of course other ways to do this, but the use of a p-value is the most common one. Other situations should be considered on a case by case basis. Response Type Yes/No Answer Criteria Yes: If the p-value identified is a number less than or equal to 0.05. No: If the p-value identified is a number greater than 0.05. Logic DISABLEMENT LOGIC - Disabled if EE02 does not contain RCT or observational OR disabled if EE05 is no. Scoring Impact There is no scoring impact associated with this question.
For each type of relevant research article: Does the P-Value demonstrate near significance (p<0.2)?	ORC_EE12
Further Information Guidance/Context ORCHA uses p-values to see if an app has demonstrated a benefit. It is possible that the app can have a benefit, but for the purposes of this one particular study it has not reached the accepted significance level, but do come close. ORCHA use this question to recognise this. Response Type Yes/No Answer Criteria Yes: If the p-value identified is a number less than or equal to 0.2. No: If the p-value identified is a number greater than 0.2. Logic DISABLEMENT LOGIC - Disabled if EE02 does not contain RCT or observational OR disabled if EE05 is yes and EE06 is no. Scoring Impact There is no scoring impact associated with this question.
For each type of relevant research article: Is there a comparator?	ORC_EE07
Further Information Guidance/Context The use of a comparator allows a comparison to be made between the app’s benefits and something else. This gives context for the benefits that the app may or may not have demonstrated Response Type Yes/No Answer Criteria Yes: If the research article/paper identified has a separate group than that which is the experimental condition. For example if the researcher is comparing against a baseline which may have come from the user prior to the intervention. The comparator could be as simple as paid version of app vs free version of an app. No: If the research article/paper identified has only the experimental condition. Logic DISABLEMENT LOGIC - Disabled if EE02 does not contain RCT or Observational. Scoring Impact There is no scoring impact associated with this question.
For each type of relevant research article: Is the comparator validated?	ORC_EE08
Further Information Guidance/Context The use of a comparator allows a comparison to be made between the app’s benefits and something else. This gives context for the benefits that the app may or may not have demonstrated. A validated comparator means a current standard treatment pathway. An example may be a depression app being compared to an antidepressant. Response Type Yes/No Answer Criteria Yes: If the research article/paper identified has a separate group than that which is the experimental condition, and the statement about that group includes “Current standard of care, Usual care or Treatment as usual”. No: If the research article/paper identified has only the experimental condition, or the paper does not mention “current standard of care, usual care or treatment as usual”. Logic DISABLEMENT LOGIC - Disabled if EE02 does not contain RCT or Observational OR disabled if EE07 is no. Scoring Impact There is no scoring impact associated with this question.
For each type of relevant research article: Has the research article been published in a Journal?	ATA_CE01
Further Information Guidance/Context If research has been published in a Journal, that research is verified as scientifically reliable. Response Type Yes/No Answer Criteria Yes: Examples of Journals which the assessor could accept: JMIR, Nature, Lancet, Digital Health, BMJ, JAMA. No: If the research has not been published in a Journal. Logic DISABLEMENT LOGIC - Disabled if EE02 does not contain RCT or Observational. Scoring Impact There is no scoring impact associated with this question.
For each type of relevant research article: Does the journal use peer-review?	ATA_CE02
Further Information Guidance/Context If research has been published in a Journal, that research is verified as scientifically reliable. If an article has been peer-reviewed it means that a board of reviewers, who are experts in the field, review the articles submitted by researchers for relevant, quality and adherence to scientific standards. Peer-review acts as a quality control mechanism. Response Yes/No Assessment Criteria Yes: Examples of leading peer-reviewed Journals which the assessor could accept: JAMA, NEJM, Annals of Internal Medicine, Lancet, BMJ, JAMA Internal Medicine, PLOS Medicine and Centers for Disease Control (esp. for infectious diseases). No: If the research has not been published in a peer-reviewed Journal. Logic DISABLEMENT LOGIC - Disabled if CE01 is no. Scoring Impact There is no scoring impact associated with this question.
For each type of relevant research article: Is the research article supplied a conference poster?	ATA_CE03
Further Information Guidance/Context Research may appear in the format of a conference poster. This provides a high level summary of the research so usually some detail around the research may not be available to the reader. Response Yes/No Assessment Criteria Yes: If the research has been summarized to fit on one page for conference purposes. No: If the research appears in any other format than a conference poster. Logic DISABLEMENT LOGIC - Disabled if EE02 does not contain RCT or Observational. Scoring Impact There is no scoring impact associated with this question.
For each type of relevant research article: Is the research article self published only?	ATA_CE04
Further Information Guidance/Context Research articles can be self-published which means he author takes the research through the editorial process themselves. This does mean that it has not gone through the quality control mechanism that a peer-review process serves. Response Yes/No Assessment Criteria Yes: If the research has been published by the author/those involved in writing the article. No: If the research has been published by a Journal. Logic DISABLEMENT LOGIC - Disabled if EE02 does not contain RCT or Observational OR disabled if CE01 is yes. Scoring Impact There is no scoring impact associated with this question.
Does the developer provide sufficient evidence that supports all the claimed benefits?	ORC_EE01
Further Information Guidance/Context This question aims to discover if there is evidence which backs up the app’s claims and intensions. The assessor should refer to what they selected in BF01 - ‘What are the claimed or implied benefits of the app?’. Response Yes/No Assessment Criteria Yes: If the evidence found supports all the benefits which you have found/ the developer has claimed. No: If the evidence found supports only some of the claimed benefits, or does not support any benefits at all. Logic DISABLEMENT LOGIC - Disabled if EE02 contains none. Scoring Impact There is no scoring impact associated with this question.
Does the developer provide sufficient evidence that supports some of the claimed benefits?	ORC_EE09
Further Information Guidance/Context This question aims to discover if there is evidence which backs up some and not all of the app’s claims and intensions. The assessor should refer to what they selected in BF01 - ‘What are the claimed or implied benefits of the app?’. Response Yes/No Assessment Criteria Yes: If the evidence found supports some of the benefits which you have found/ the developer has claimed. No: If the evidence found supports none of the claimed benefits, or does not support any benefits at all. Logic DISABLEMENT LOGIC - Disabled if EE02 contains none OR EE01 is yes. Scoring Impact There is no scoring impact associated with this question.

Behavioral Change

There are some scenarios where the app utilizes widely accepted techniques with a breadth of evidence. In this instance, the app may not deem it appropriate to fund a full randomized control trial to demonstrate effectiveness. Therefore we give some value points for fully referencing evidence for behavioral change techniques used within the app. This is not however treated in the same way as to where the app has provided direct evidence of its own effectiveness.

Question Set

Does the app have its own high quality study?	ORC_BCT01
Further Information Guidance/Context The purpose of this question is to identify evidence that the app has performed its own study on behavior change techniques, which meets the ESF requirements of the tier. This is information gathering, and is more important for the following question, BCT02. Response Type Yes/No Answer Criteria Yes: If the research article/paper identified is suitable evidence for an app of that ESF tier. For example, if the app is tier Cii on the NICE ESF, and the evidence identified is an RCT, with a significant p value and validated comparator. Tier Ci - Needs to be minimum of an observational study with a comparator and a significant P value/confidence interval. Tier Cii - needs to be an RCT with significant P Value/Confidence Interval and validated comparator. Any tier below Ci, the app would only need to tick off the Ci requirements to answer this question yes. No: If the research article/paper identified does not have a high quality study, suitable for an app at that tier. For example, the app is tier Cii on the NICE ESF, but the evidence identified is an observational study but ongoing Logic DISABLEMENT LOGIC - Disabled if TS11 does not contain CBT or Preventative Behavior Change. Scoring Impact There is no scoring impact associated with this question.
Does the app reference and evidence its behavior change technique?	ORC_BCT02
Further Information Guidance/Context This question is to differentiate those apps which don’t have a study. If the app has not got its own study and they use a behavior change technique, then this question looks to see if it is referenced. If it is referenced then this allows a small increase in value to the scoring, even though the developer themselves may not have the necessary study demonstrating efficacy of the specific product. Response Type Yes/No Answer Criteria Yes: If the developer displays research on which the app is based. For example, an app may have built a feature into their app based on other research, or they may refer to a paper about the psychological intervention it is based on, eg CBT, “we added these features based on this paper” etc. No: If the developer does not reference or evidence the behavior change the app is based on OR if they mention briefly, but don’t provide specific links. Logic DISABLEMENT LOGIC - Disabled if TS11 does not contain CBT or Preventative Behavior Change OR disabled if BCT01 is yes. Scoring Impact Medium - High value applied if yes. More value applied if the app is Cii ESF Tier.

Professional Backing

We look for evidence of an appropriate professional being involved in the app's design and development, or if the app has been externally accredited. A relevant professional is deduced in the context of an app. For example, for a simple yoga app, we would accept a qualified yoga instructor as a relevant professional, but for a complex clinical solution, we would only accept a relevant qualified physician. External accreditations are wide-ranging, but we would look for an appropriate body, for example, the American Heart Association (ATA) giving an endorsement to a cardiology app.

Question Set

Is there a suitably qualified Professional involved in the Development team of the App?	ORC_PB01
Further Information Guidance/Context This question looks to identify if there was a relevant professional part of developing the app, this helps indicate that the information contained within is relevant. Response Type Yes/No Answer Criteria YES: If there is evidence of a suitably qualified professional being involved with the app. For example, a CBT website displays a psychologist on the “Our team” page of their website. NB It is important to attempt validate this named professional. No: If the developer does not reference a suitably qualified professional OR it is not clear what role they play. For example, a psychologist is named, but it is unclear whether they simply use the app, or where involved in the development. Logic There is no disablement logic written for this question. Scoring Impact Varying value based on ESF tier of app. Value is not awarded for both PB01 and PB02.
Who was the suitably qualified Professional involved, and what are their qualifications?	ATA_PB02
Further Information Guidance/Context This question looks to identify if there was a relevant professional part of developing the app, this helps indicate that the information contained within is relevant. Response Type Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if PB01 is no. Scoring Impact There is no scoring impact associated with this question.
Does the organization behind the App have relevant credentials?	ORC_PB02
Further Information Guidance/Context This question generally looks to assist larger organizations that may not have the ability or practicality to name individuals involved in the creation of the app. Response Type Yes/No Answer Criteria Yes: If the app is made by an institution that is believed to have the relevant experience. For example, the app is produced by the CDC. No: If the app does not have any relevant credentials, and was simply produced by a development company. Logic There is no disablement logic written for this question. Scoring Impact Varying value based on ESF tier of app. Value is not awarded for both PB01 and PB02.
Who was the recognized or national health body involved in the development of the app?	ATA_PB03
Further Information Guidance/Context This question generally looks to assist larger organizations that may not have the ability or practicality to name individuals involved in the creation of the app. Response Type Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if PB02 is no. Scoring Impact There is no scoring impact associated with this question.
Is there evidence of an endorsement by a relevant body?	ORC_PB03
Further Information Guidance/Context This question helps provide assurance that the app works as intended from any endorsements that can be seen from relevant healthcare bodies. Response Type Yes/No Answer Criteria YES: If there is evidence of any sort of accreditation by any sort of relevant organization, or professional body. For example, a Diabetes app with an endorsement from the National Diabetes Foundation. NICE, FDA approval and ISO 13485 would be sufficient here. No: If the app does not have any relevant endorsements OR if the endorsement is from an individual, rather than an organization or body OR if the organization endorsing the app are in some way involved with the development/content of the app. Logic There is no disablement logic written for this question. Scoring Impact High value applied if yes.
Who are the relevant body who have endorsed the app?	ATA_PB04
Further Information Guidance/Context This question helps provide assurance that the app works as intended from any endorsements that can be seen from relevant healthcare bodies. Response Type Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if PB03 is no. Scoring Impact There is no scoring impact associated with this question.
Has the app been adopted by a relevant healthcare organization within the US?	ATA_PB01
Further Information Guidance/Context This question helps provide assurance that the app works as intended if healthcare organizations have adopted the app and are using the app. Response Type Yes/No Answer Criteria Yes: If the on the app’s website there are statements that the app has been adopted and used within X organization. Also yes if an organization has promoted their app on their own website and stated they have adopted it within the organization. Web articles stating an organization's adoption of use would also be acceptable. No: If there’s no evidence publicly available which states the app has been adopted within a relevant healthcare organization. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Which US healthcare organization had adopted the app?	ATA_PB06
Further Information Guidance/Context This question helps provide assurance that the app works as intended from any endorsements that can be seen from relevant healthcare bodies. Response Type Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if PB03 is no. Scoring Impact There is no scoring impact associated with this question.
Are organizations using the app?	ORC_PB04
Further Information Guidance/Context This question helps provide assurance that the app works as intended if healthcare organizations have adopted the app and are using the app. Response Type Yes/No Answer Criteria Yes: If there is evidence (can be a statement) of any sort of relevant organization using the app. For example, the website may display that the platform is used by a CCG, or display the relevant logos. No: If it is not clear any organizations are using the app OR if a person references their position in an organization, but doesn’t make it clear it is organizational use. Logic There is no disablement logic written for this question. Scoring Impact Varying value if yes based on ESF tier of app.
Which relevant organization are using the app?	ATA_PB05
Further Information Guidance/Context This question helps provide assurance that the app works as intended if healthcare organizations have adopted the app and are using the app. Response Type Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if PB04 is no. Scoring Impact There is no scoring impact associated with this question.
Is there a statement that it has been positively evaluated or validated by a relevant healthcare professional?	ORC_PB05
Further Information Guidance/Context This question helps provide assurance that if a healthcare professional is willing to positively evaluate an app, using their own name/qualifications, it provides assurance that the app works as expected. Response Type Yes/No Answer Criteria Yes: If there is evidence of any sort of testimonial or accreditation by any sort of relevant individual (external from company). For example, a Diabetes app accredited by a Diabetologist. No: If the app does not have any relevant endorsements OR if the endorsement is from an organization or body, rather than an individual. Logic There is no disablement logic written for this question. Scoring Impact Varying value based on ESF tier of app.
Please specify who the relevant experts are and what qualifications they hold.	ORC_AE17
Further Information Guidance/Context For data collection purposes, please record who the relevant expert is. Where possible the qualifications of the professional should be validated. Response Type Free text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if PB05 is no. Scoring Impact There is no scoring impact associated with this question.
Is there evidence within the app that the developer has validated any guidance with relevant reliable information sources or references?	ORC_PB06
Further Information Guidance/Context The point of this question is to establish whether the information provided comes from a relevant and reliable source (can be a qualified person/organization/citation of original journal article). Response Type Yes/No Answer Criteria Yes: If a link to a source is provided - even if the link can’t be clicked, if you can type it in and it’s valid then yes OR if the Developer uses a well established tool, which they reference (GAD-7, PHQ-9 etc) OR if the Developer links to external information, which comes from a reputable outside source (NHS choices, PHE etc.). No: If the app does not have any relevant guidance which has been validated, either in the form of references, or using cleared named clinical calculators. Logic There is no disablement logic written for this question. Scoring Impact Varying value based on ESF tier of app.
Is there any evidence within the app that the developer has validated the information, advice or guidance with relevant and appropriate academic studies or relevant academic expert input?	ORC_AE03
Further Information Guidance/Context The point of this question is to establish whether the information provided comes from a relevant and reliable source (can be a qualified person/organization/citation of original journal article). US-centric Sources for clinical evidence and professional credibility: Evidence-based Guidelines:  American College of Physicians  US Preventive Services Task Force  Canadian Task Force on Preventive Health Care  American Diabetes Assn.  American Heart Assn.  American Cancer Society  Am. Psychological Assn.  American College of Obstetrics and Gynecology Response Yes/No Assessment Criteria YES: If a link to a source is provided - even if the link can’t be clicked, if you can type it in and it’s valid then yes OR if the Developer uses a well established tool, which they reference (GAD-7, PHQ-9 etc) OR if the Developer links to external information, which comes from a reputable outside source eg Websites sponsored by Federal Government Agencies or well known medical schools (CDC, Mayo Clinic etc.). NO: If the app does not have any relevant guidance which has been validated, either in the form of references, or using cleared named clinical calculators Logic There is no disablement logic written for this question. Scoring Impact Varying value based on ESF tier of app.

ORCHA Adapted ESF Compliance

The first part of this section assesses which ESF Tier the app falls under, and is non-scoring. The second part assesses whether the app meets the requirements of that Tier. Compliance with the ESF is determined by the app answering positively to all questions that have been flagged as a requirement for its Tier of the ESF and all Tiers below.

Question Set

Is the app Tier A?	ORC_ESF01
Further Information Guidance/Context Classification for tiering of an app is defined by the scene setter questions asked to help assess the functions and features of an app. Response Type Yes/No Answer Criteria Yes: To be classified as a Tier A the app must: Have met none of the requirements for any other tier & provides no patient outcomes. E.g. be an administration app. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Is the app Tier Bi?	ORC_ESF02
Further Information Guidance/Context Classification for tiering of an app is defined by the scene setter questions asked to help assess the functions and features of an app. Response Type Yes/No Answer Criteria Yes: To be classified as a tier Bi app it must- Provide information or guidance/context (I01 is yes). OR Allow a healthcare professional to provide clinical advice, as opposed to the app providing it (EF09 is yes). OR Provide information, resources, or activities to the public, patients, or clinicians, either about a specific condition or general health and lifestyle (EF07 is yes). OR Provide two-way communication between patients, citizens or healthcare professionals (EF10 is yes). OR The app is a simple self-management app (selected in MN04). Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Is the app Tier Bii?	ORC_ESF03
Further Information Guidance/Context Classification for tiering of an app is defined by the scene setter questions asked to help assess the functions and features of an app. Response Type Yes/No Answer Criteria Yes: To be classified as a tier Bii app it must- Do none of the things listed in Ci/Cii & it is a standard self management app as defined by the scene setter questions (MN04). Example: If an app is simple monitoring with a specific condition focus OR complex monitoring with a wellbeing and general health focus, as defined by the scene setter questions. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Is the app Tier Ci?	ORC_ESF04
Further Information Guidance/Context Classification for tiering of an app is defined by the scene setter questions asked to help assess the functions and features of an app. Response Type Yes/No Answer Criteria Yes: To be classified as a Tier Ci app it must- Have no things listed in Cii AND Be a complex self management app (selected in MN04). OR Have preventative behavior change within the app (selected in TS11). OR Have a recognized (not novel) clinical calculator within the app (TS01 is yes and TS02 mentions an established clinical calculator). Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Is the app Tier Cii?	ORC_ESF05
Further Information Guidance/Context Classification for tiering of an app is defined by the scene setter questions asked to help assess the functions and features of an app. Response Type Yes/No Answer Criteria Yes: To be classified as a Tier Cii app it must- - Diagnose a condition (DG02 is yes). OR - Have a novel clinical calculator within the app which impacts care, treatment, or diagnosis (TS01 is yes and TS02 mentions a novel clinical calculator). OR - Automatically measures and/or records data about a user’s specified condition, and transmits the data to a professional, carer, or third party organization, without any input from the user (MN07 is yes). OR - Provide treatment (TS05 is yes). OR - Guide the treatment of a condition (TS07 is yes). OR - Alleviate the symptoms of an existing condition (TS15 is yes). Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Tier A requirements?	ORC_ESF06
Further Information Guidance/Context The app has met Tier A requirements if the app has: -Evidence of a survey, pilot study, meta-analysis, RCT, observational, or other indicated user acceptance/benefit (EE02 does not contain none). AND at least one of the following has been answered yes: - Evidence of a relevant professional involved in the development team (PB01). - Relevant organizational credentials (PB02). - Evidence of endorsement by a relevant body (PB03). Response Type Yes/No Answer Criteria Yes: If the app has fulfilled the requirements listed in the guidance. Logic There is no disablement logic written for this question. Scoring Impact High risk if no or high value if yes + multiplier for value applied based on evidence available.
Tier Bi requirements?	ORC_ESF07
Further Information Guidance/Context The app has met tier Bi requirements if the app has all of the following criteria: - Evidence that the developer has validated the information, advice or Guidance/Context with relevant academic expert input (PB01 or PB02 or PB06 is yes). AND - There is clear evidence of safeguarding measures being in place for any communication functions (AE13 is yes, if applicable). AND - The app has evidence of accrediting expertise (PB01 or PB02 or PB03 or PB05 is yes). Response Type Yes/No Answer Criteria Yes: If the app has fulfilled the requirements listed in the guidance. Logic There is no disablement logic written for this question. Scoring Impact High risk if no or high value if yes + multiplier for value applied based on evidence available.
Tier Bii requirements?	ORC_ESF08
Further Information Guidance/Context The app has met tier Bii minimum requirements if the app has: - Evidence that the developer has validated the information, advice or Guidance/Context (PB01 or PB06 is yes). AND - Clear evidence of safeguarding measures being in place for any communication functions (AE13 is yes, if applicable). AND - Evidence of accrediting expertise (PB01 or PB02 or PB05 is yes). AND - Evidence of an endorsement by a relevant body (PB03 is yes). OR - A meta-analysis, or an observational study/RCT with a p-value < 0.05 (EE02 is a yes AND one of the EE06 answers is a yes). NB - If an app has met tier A, tier Bi AND tier Ci requirements, then the app will have met Bii requirements. Response Type Yes/No Answer Criteria Yes: If the app has fulfilled the requirements listed in the guidance. Logic There is no disablement logic written for this question. Scoring Impact High risk if no or high value if yes + multiplier for value applied based on evidence available.
Tier Ci requirements?	ORC_ESF09
Further Information Guidance/Context The app has met Tier Ci requirements if it has: Evidence of an RCT (EE02 answer includes RCT) which has a significant p value (EE06 is yes) OR Evidence of an observational study (EE02 answer includes observational) which has a significant p value (EE06 is yes) AND A comparator (EE07 is yes) OR a validated comparator (EE08 is yes). Response Yes / No Assessment Criteria Yes: If the app has fulfilled the requirements listed in the guidance. Logic There is no disablement logic written for this question. Scoring Impact High risk if no or high value if yes + multiplier for value applied based on evidence available.
Tier Cii requirements?	ORC_ESF10
Further Information Guidance/Context The app has met Tier Cii minimum requirements if it has: - Evidence of an RCT (EE02 answer includes RCT) which has a significant p value (EE06 is yes). AND - A validated comparator (EE08 is yes). Response Type Yes/No Answer Criteria Yes: If the app has fulfilled the requirements listed in the guidance. Logic There is no disablement logic written for this question. Scoring Impact High risk if no or high value if yes + multiplier for value applied based on evidence available.
Does the app have appropriate evidence for the ESF tier?	ORC_ESF11
Further Information Guidance/Context Use the above questions as a guide to determine the answer. If no, provide an explanation why. Response Type Yes/No Answer Criteria Yes: If the app has met its own tier, plus those below, as the requirements are cumulative. No: If the app has not fulfilled the requirements of its own tier. No: If the app has met only the requirements at its own tier and not those below. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.

	Functionality or Purpose of DHT	Examples	Assurance of quality provided by the following (described in more detail below this chart)
Tier A	No measurable patient impact but provides services to health systems.	App based roster services Apps which provide electronic versions of mental health forms.	Evidence that a suitably qualified professional or professional body is behind the DHT
Tier Bi	Provides information and guidance to the user. Allows a healthcare professional to provide advice via the app. Allows two-way communication between patients or healthcare professionals. Simple monitoring, defined as recording data over time for the user to review, with no “intelligent” manipulation of the data by the app, and where the monitoring is not condition-specific. App does not interpret the data to produce a reminder, alert or notification.	Self-monitoring with a focus on well-being and general health. App-based journal to log symptoms or other markers. App may render the data in graphical views that illustrate trends and patterns.	Evidence that a suitably qualified professional or professional body is behind the DHT.
Tier Bii	Provides condition-specific simple monitoring, or App which is not condition-specific but provides the automated assessment or interpretation of relevant data to deliver alerts, insights, reminders or adjustments.	Activity tracker with reminders.	Evidence that the DHT has been adopted and used (within a recognized healthcare setting if applicable) or Evidence that the DHT is actively endorsed by a recognized healthcare system. Safety can be demonstrated through suitable test data if applicable. For clinical efficacy, a high quality observational study could be provided.
Tier Ci	Involves the automated assessment or interpretation of relevant data to deliver alerts, insights, reminders or adjustments regarding the management of a specific condition. Delivers preventative behavior change. Has a recognized clinical calculator within the app (i.e. novel clinical calculators are not included here).	DHT which logs your manually entered blood sugar levels, and provides insights regarding whether you are controlling your diabetes better or worse than last month.	Safety can be demonstrated through suitable test data if applicable i.e. accuracy of clinical calculators etc. or through high quality observational studies/RCT’s that demonstrate safety. For clinical efficacy in Tier Ci, a high quality observational study is the minimum required. High quality study indicated by: Presence of a suitable comparator and p value of <0.05 (or non-overlapping confidence intervals). Suitable sample group/size. Alignment to claimed benefits/safety risks.
Tier Cii	Diagnoses a condition, Contains a novel clinical calculator, Automatically measures/records and transmits specific condition data to a 3^rd party without input from the user, Provides treatment or guides the treatment of a condition, Alleviates symptoms of an existing condition.	Medical devices as classified by the FDA.	Safety can be demonstrated through suitable test data, or through high quality observational studies/RCT’s that demonstrate safety. Default expectation around clinical efficacy is an RCT - although there are scenarios where an RCT might not be practical, including diagnostics where it would not intuitively make sense to randomize to one diagnostic or another, but rather compare the same patients (and diagnoses) using the old (and new) diagnostics and measuring agreement between the two. High quality study indicated by: Presence of a suitable comparator and p value of <0.05 (or non-overlapping confidence intervals). Suitable sample group/size. Alignment to claimed benefits/safety risks.

Medical Devices

Question Set

Has the app been FDA approved? (Food and Drug Administration)	ORC_FDA01
Is the product a medical device, as defined by the FDA?	ATA_MD01
Further Information Guidance/Context The question aims to identify if the app in question is a medical device based on the FDA guidance. If the app is intended to prevent (MD01), compensate (MD07), monitor (MN05 AND MN08) or Alleviate (TS15) an illness, injury or handicap. Also, if the App diagnoses (PD01 AND DG02 are yes, and DG05 is yes and DG09 is no OR DG06, DG07 and DG08 are yes), PD01 and DG02 and DG05 yes, DG09 no OR DG06, DG07 and DG08 yes or treats (TS10 is yes OR TS11 contains Self Management OR TS14 is yes AND TS03 is yes and TS12 and TS13 are No) an illness, disease or handicap. TS10 yes OR TS11 contains self management OR TS14 & TS15 are yes, and TS12 & TS13 no If the App is intended to control conception (MD08 is yes, CC01 and CC02 are yes, and CC03 OR CC04 is yes). MD08, CC01, CC02 AND CC03 yes MD08, CC01, CC02 AND CC04 Response Type Yes/No Answer Criteria Yes: If the app fulfills the above guidance and is a medical device. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Further Information Guidance/Context The question is looking for if the FDA has approved a premarket approval (PMA) application, or a Humanitarian Device Exemption (HDE) application. This is for class III medical devices (highest risk) and involves a more rigorous review than the 510(k) review process. Response Type Yes/No Answer Criteria YES: If an app has been APPROVED by the FDA. Logic There is no disablement logic written for this question. Scoring Impact High value applied if yes AND DE01 contains Other or None.
Has the app been FDA cleared?	ORC_FDA02
Further Information Guidance/Context FDA CLEARANCE means that an app uses a feature/algorithm which itself has been FDA approved, and the app has been cleared to use the same feature which functions as it should do. Response Type Yes/No Answer Criteria YES: If an app has been CLEARED by the FDA. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.

Safety/Risk Management

It is proposed that the assessment looks for any safeguarding measures in the communication functions of the app, if relevant.

Question Set

Is there a statement or any evidence showing that appropriate safeguarding measures are in place around peer-support and other communication functions within the platform? (Tier Bi requirement - Only asked of apps that require such measures because of it functional capabilities / intended purpose )	ORC_AE13
Further Information Guidance/Context This question is a Tier Bi and above requirement. It is only asked of apps that require such measures because of the functional capabilities/intended purpose of the app. Response Type Yes/No Answer Criteria YES: If there is an internal forum, the content is moderated, or guidelines are set out, or it is monitored OR there is a full policy in place specific to a forum OR if two-way communication occurs and the data is protected/encrypted OR if there is a registration process where you use a HCP number OR if the communication is made clear it is only with a registered HCP. Logic DISABLEMENT LOGIC - Disabled if there are no communication functions on the app OR disabled if there is no way for a HCP to access user data. Scoring Impact High- Medium value applied if yes based on the ESF tier the app’s classified as. Varying risk value applied if no based on the ESF tier the app’s classified as.
Does the Developer clearly identify who the app should be used by?	ORC_S01
Further Information Guidance/Context The question is looking for a statement or other evidence showing who the app is intended for. Or if there are any demographics who should not use it. Response Type Yes/No Answer Criteria YES: If a developer tells us who the app SHOULD or SHOULD not be used by. Can be specific or general e.g. for 18years +, for anyone who undertakes physical activity, etc. No: If the app does not tell us who the app SHOULD or SHOULD not be used by. Logic There is no disablement logic written for this question. Scoring Impact Low value applied if yes.
Does the Developer clearly identify who the app should not be used by?	ATA_CS01
Further Information Guidance/Context The question is looking for a statement or other evidence showing who the app is not intended for. Or if there are any demographics who should not use it. Response Type Yes/No Answer Criteria YES: If a developer tells us who the app SHOULD NOT be used by. Can be specific or general e.g. not appropriate for anyone under 16 years. No: If the app does not tell us who the app SHOULD be used by. Logic There is no disablement logic written for this question. Scoring Impact Low value applied if yes.
Does the Developer publish their clinical risk management processes?	ORC_S02
Further Information Guidance/Context It is understood that risk management documents may contain company sensitive information. Therefore documents do not have to be made publicly available but could be made available upon request, or there could be a detailed explanation for the process involved within the developers risk management process. Response Type Yes/No Answer Criteria YES: If there is evidence of a risk management process. This may be in the form of a hazard log or safety case, and will likely be made available through the website, if available at all. No: If the developer does not clearly display their risk management processes. Logic There is no disablement logic written for this question. Scoring Impact Low value applied if yes.
Does the Developer make clear risks associated with using the app?	ORC_S03
Further Information Guidance/Context This question provides context to the user to make an informed decision about the risks associated with the app, and whether a user would still want to use it. Response Type Yes/No Answer Criteria YES: If the developer defines clearly what possible risks there are to a user - this may be in the form of a hazard log or safety case. A disclaimer highlighting the risks. No: If the developer does not clearly display the risks associated with their app. A disclaimer highlighting that information in the app is not medical advice or something along those lines is not suitable to meet these requirements. Logic There is no disablement logic written for this question. Scoring Impact Low value applied if yes.
Is there a way for the user to confirm that the data input is accurate?	ORC_S04
Further Information Guidance/Context This question is looking to identify what validation is applied to data inputted by the users. It is looking to see if the app checks for erroneous data, this helps ensure the safety of the app by not allowing mis-calculations from inputted data. Response Type Yes/No Answer Criteria YES: If data is being entered the app requires confirmation. For example if a users was to input 5000 mmol/l for glucose readings, does it ask for confirmation? If the app uses sliders to restrict data entry parameters then this would also be a yes. Should be noted that this should somehow be related to the app function, rather than other data input. No: If the developer does not ask the user to confirm input. Logic DISABLEMENT LOGIC - Disabled if MN01 contains None. Scoring Impact Low risk applied if no + multiplier based on functionality complexity.
Does the app have a clinical person responsible for patient safety risks?	ATA_CS02
Further Information Guidance/Context A person responsible for the management of the clinical risk processes ensuring that any patient safety risk have been mitigated and appropriate controls and safety procedures are being followed. Response Type Yes/No Answer Criteria YES: If there is evidence of responsible person, anywhere within the app or associated sites/documentation. The responsible person should have the relevant experience and expertise for example, previously or currently qualified as a Doctor, or Psychologist etc. No: If the developer does not clearly name a responsible person acting in this position. No: If the responsible person does not have a relevant background to account for patient safety risks. Logic There is no disablement logic written for this question. Scoring Impact Medium risk applied if no.
Is there a way for users to submit safety concerns?	ATA_CS03
Further Information Guidance/Context A way for users to submit concerns they have around the safety of the app. This is different from the user being able to submit technical issues. Response Type Yes/No Answer Criteria YES: If the developer specifically states that the user can submit safety concerns. They would also need to provide an email address, an eTicket service or a another form of contact method for safety concerns. No: The developer only makes available a process where the user can submit technical issues. Logic There is no disablement logic written for this question. Scoring Impact Low value applied if yes.

USABILITY AND ACCESSIBILITY

Design and Development

This considers the design and development of the app and whether it follows any recognized app design standards, such as WC3, WCAG 2.0 AA, WCAG 2.1 AA, ISO 9241, Apple HIG, or Android App Quality Guidelines. The review also considers whether there was any user involvement during the development of the app, user involvement in testing, or if any features were based on user feedback.

Question Set

Is there a statement about user feedback during design/development?	ORC_DT01
Further Information Guidance/Context This question is to determine if relevant users/user feedback have been considered, in the design of the app - BEFORE or AFTER the app was released. Response Type Yes/No Answer Criteria YES: If the developer has added features based on user feedback, and states what has changed. This can be before or after the app was published, but changes must have been made. If the app was designed by the developer to remedy a problem they were suffering, or caring for someone suffering. If an app is developed by doctors, for doctors. If the app makes changes based on data collected, or users updating database e.g. MyFitnessPal. If the app has undergone a survey/pilot study involving users, and changes were made based on the outcomes. NO: If the app states “may add features based on feedback” - it needs to state which specific features were added. Logic There is no disablement logic written for this question. Scoring Impact Low value applied if yes.
Is there any evidence of user involvement in testing?	ORC_DT02
Further Information Guidance/Context This question is to determine if there is any evidence that users have tested, or provided feedback on the app AFTER it was released. Response Type Yes/No Answer Criteria YES: If there is a case study on the developer website. If there was a Beta version of the app available before the app went live. If user feedback is shown on the website showing the app has been beneficial to users (e.g. 87% of patients have shown improvement from using the app). Any other evidence of user testing rather than opinions from the general public. Any evidence of indicated user benefit (if you have selected this in EE02). No: If the only user feedback is from app store reviews. Logic DISABLEMENT LOGIC - Disabled if CUS01 AND CUS02 are no. Scoring Impact Medium value applied if yes.
Was there evidence of a cross-section of society included?	ATA_UA01
Further Information Guidance/Context This question aims to establish whether a wide range of users/user groups/potential users were considered within the design and development of the product. If user testing was carried out, was it carried out with a cross-section of society, rather than a select or pre-defined group of users? This is to ensure that multiple perspectives were considered in the design and development of the product. Response Yes / No Assessment Criteria YES: Evidence that testing has been done with a good sample of intended users - e.g. various age groups, communities, conditions (if app is not condition specific) or co-morbidities. Logic DISABLEMENT LOGIC - Disabled if DT01 and DT02 are no. Scoring Impact
Is there any evidence that user feedback is considered?	ATA_UA02
Further Information Guidance/Context It’s important that developers respond to user feedback, and continuously update/improve their product based on this feedback, for the user’s benefit. It’s one thing to receive user feedback - this question is to establish what the developer intends to do with it, and if the user feedback is reflected in future updates. Response Yes / No Answer Criteria Yes: Evidence that relevant user comments and feedback are considered during updates. Yes: Evidence that user complaints are recorded and responded to. Logic There is no disablement logic written for this question. Scoring Impact Medium value applied if yes.

Accessibility

Accessibility is important to consider, as the app should be accessible to all users regardless of their specific needs. The review considers whether the app is customizable to suit certain needs, such as poor sight or hearing impairments. If the app uses any specialist or medical terms, these should be clearly explained to the user.

Question Set

Is there a statement within the app outlining compliance with any currently recognized app design standards? WC3 · WCAG 2.0 AA · WCAG 2.1 AA · ISO 9241 · Apple HIG · Android App Quality Guidelines	ORC_DE01
Further Information Guidance/Context This information is likely to be found in the accessibility statement, it may also be found in the about section within the app or on the developer website. Choose from the available options, or click none if none apply. Response Type Multiple Choice Answer Criteria WC3 WCAG 2.0 AA WCAG 2.1 AA ISO 9241 Apple HIG Android App Quality Guidelines Other (please specify) None Logic There is no disablement logic written for this question. Scoring Impact High value applied if WC3, WCAG 2.0 AA, WCAG 2.1 AA, ISO 9241, Apple HIG or Android App Quality Guidelines is selected.
Can the user change the font size in-app/does the app respond to device preferences?	ORC_U04
Further Information Guidance/Context This is a key aspect for improving accessibility of apps to demographics with accessibility needs. Response Type Yes/No Answer Criteria YES: The app responds to font size changes in the device, or the font size can be changed from within the app. Logic DISABLEMENT LOGIC - Disabled if CUS01 AND CUS02 are no. Scoring Impact Low value applied if yes.
Does the app provide support for users with poor sight?	ORC_U07
Further Information Guidance/Context This question aims to address whether the app developer has considered the accessibility needs of people with perceptual impairments, specifically poor sight, e.g. blind, color blind, poor vision. Response Type Yes/No Answer Criteria YES: If the app provides audio description, or visual descriptions of pictures. If the app uses Voice Over (iOS) or Text to Speech (Android). If there is the ability to change the font size, or zoom in, or make color adjustments. Logic DISABLEMENT LOGIC - Disabled if CUS02 are no. Scoring Impact Medium value applied if yes.
Does the app provide support for users with hearing difficulty?	ORC_U08
Further Information Guidance/Context This question aims to address whether the app developer has considered the accessibility needs of people with perceptual impairments, specifically poor hearing, e.g. deafness, or hard of hearing. Response Type Yes/No Answer Criteria YES: If the app provides audio description and you can adjust the volume of Text to Speech/voiceover (if using in-built text to speech on iOS and Android the volume can be adjusted) If subtitles are available for video/audio/in game dialogue Logic DISABLEMENT LOGIC - Disabled if CUS02 are no. Scoring Impact Medium value applied if yes or medium risk applied if no.
Has the product been through FDA approval and therefore meets FDA Usability guidelines?	DHAF_UA04
Further Information Guidance/Context The DHAF is interested in whether the app has been FDA approved, as there are many usability standards in this approval process. FDA approval would therefore give a reasonably confident view of the app’s usability. Response Type Yes/No Answer Criteria YES: If the app provides audio description and you can adjust the volume of Text to Speech/voiceover (if using in-built text to speech on iOS and Android the volume can be adjusted) If subtitles are available for video/audio/in game dialogue Logic Scoring Impact

Usability

This also ties into the usability of the app, including further customization options. The review identifies if the app has any functions to aid navigation, such as a home button, back button, help button or search feature. If the app utilizes push or email notifications, the review identifies whether the user has options to manage these for their own preference or privacy, both at the app level and at the device level. Finally, if there are any bugs identified during the review, this will be flagged. If the app contains a forum, then we look for a statement to ensure that forum content is moderated

Question Set

Can the user change the presentation theme?	ORC_U06
Further Information Guidance/Context This question is looking to see if the app developer has considered accessibility needs for a breadth of audiences, i.e. not specific to certain impairments. Or if they have considered the usability/customization of the product. If users are able to tailor an app to their own preferences/needs will increase usability and accessibility - for example, if they can change the language to their preferred language, change the units to something more understandable, or change the color scheme to something easier on their vision. Response Type Yes/No Answer Criteria YES: If any visible changes can be made, which are not otherwise mentioned. I.e. the ability to change colors, profile pictures, language, units, music etc. Logic DISABLEMENT LOGIC - Disabled if CUS01 are no. Scoring Impact Low value applied if yes.
Does the app include the following functions: · Home/Menu button · Back button · Help/About button · Search button	ORC_U32
Further Information Guidance/Context By having familiar buttons such as home/help/search/about users can more easily navigate the app, as users will be familiar with things such as a magnifying glass representing the search feature. Response Type Multiple Choice Answer Criteria Home: A button from any page back to the original page the app opens on. This should be accessible from all pages. Back: The ‘back’ on an android phone does not count, it must be in-app. Help: A tutorial, or how to use the app or certain features. Search: A search bar, or any other way to filter and find information. None Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Are any medical, specialist or technical terms explained clearly to the user?	ORC_U15
Further Information Guidance/Context This question is to improve accessibility of the product, ensuring the app developer has considered the needs of users who may have a lower digital literacy, a lower reading level, or a lack of specialist knowledge. Explaining all key terms improves the accessibility of the product, regardless of knowledge level. Response Type Yes/No Answer Criteria YES: An instruction of how to do an exercise, even if it’s a picture. A glossary, or any definition of specialist terms (only 1 definition is needed). Logic DISABLEMENT LOGIC - Disabled if no information is provided on the app, the app does not constitute as a medical device and if no health/wellness monitoring occurs on the app. Scoring Impact Medium value if answered yes.
Does the user have options to manage the notification settings (push/email) within the app for convenience/privacy?	ORC_D31
Further Information Guidance/Context This functionality would allow users to control and manage their notifications which increases the likeability/usability of the product. Apps which show notifications/pop ups with sensitive information may not be preferable to a user, if they have no way to disable this. Additionally, it can be simply annoying. Response Type Yes/No Answer Criteria YES: If there is the ability to toggle notifications, or choose the time they are sent from within the app. Logic DISABLEMENT LOGIC - Disabled if D29 is no and D30 is no. Scoring Impact Low value applied if answered yes.
Does the app inform the user how to manage notification settings for convenience/privacy (to prevent info being shown if device is locked but on show)?	ORC_D32
Further Information Guidance/Context This is related to how a user can control notifications via the device settings. The previous question is within the app, this question is focused at the device level. This helps ensure that no information is shown on the lock screen that may be private to the user. iOS does this by default with the notifications pop-up that appears upon initial opening of an app so the purpose of this question is to target apps which are downloadable on android platforms. Response Type Yes/No Answer Criteria YES: If you can control the privacy of notifications. This is almost always yes for iOS (if it sends a pop-up), and almost always no for android. YES: Android - If they provide instructions of how to disable notifications within the device settings Logic DISABLEMENT LOGIC - Disabled if D29 is no and D30 is no. Scoring Impact Low value applied if answered yes.
Was there any evidence of bugs during review?	ORC_U23
Further Information Guidance/Context If a bug is identified it should be assessed by another person/device to confirm. If a bug is confirmed then the Developer should be notified. Response Type Yes/No Answer Criteria YES: If the app crashes or shuts down. If a link leads to the wrong place. If specific buttons don’t work. Logic There is no disablement logic written for this question. Scoring Impact High risk applied if yes.

Support

Support is a key area of this section, as it is important that users are informed of ways in which they can contact the developer should they have any problems or questions with the app. The review also identifies what type of support is offered to users, and if there is a commitment from the developer to respond to any user queries. We would expect to see the type of support offered is appropriate to the app level - a higher level app would therefore require a more sophisticated offer of user support.

Question Set

If there is a forum, is there a statement within the app that the forum content is moderated?	ORC_FC03
Further Information Guidance/Context If there is a forum, or any peer communication between users, it is important that there is moderation, guidelines or safeguards in place, to protect users from harmful or incorrect content. This introduces an element of safety to ensure the users aren’t exposed to false information in relation to their health. Many users will feel more comfortable using forums if they know the content is moderated. Response Type Yes/No Answer Criteria YES: If there is mention of moderation, community guidelines, or if users are asked to report offensive material. If there is mention of the developer reserving the right to modify or remove content. NO: If there is a statement about the risks of following third party links and no mention of the above. Logic DISABLEMENT LOGIC - Disabled if there is no internally hosted forum or community. Scoring Impact Medium risk applied if yes.
Is there a statement about how to report issues to the developer?	ORC_U24
Further Information Guidance/Context Users should be able to raise any issues easily to developers. This can be identified either within the app or on the developer website. It needs to be clear the details for contacting are about the app or the website is for the app only. Response Type Yes/No Answer Criteria YES: If a contact method is provided within the app, or accompanying website. If there is any way for the user to contact the developer electronically from the app. NO: If the only way to contact is to provide an app store rating. Email addresses provided as standard on the Play Store do not qualify. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Is there a statement about how users can raise a query about their healthcare?	ATA_UA07
Further Information There should be different types of support depending on the query, so the user is directed to the right person. For instance, technical queries can often be addressed by the developer whereas healthcare queries should be addressed by a relevant professional. Response Yes / No Assessment Criteria YES: If the app or website provides detail or directs the user to suitable support in relation to their healthcare or condition. NO: If the only support the app offers is technical/in relation to the app If the app states, for example, ‘we are not qualified to provide medical advice, please consult your physician’. Logic There is no disablement logic written for this question. Scoring Impact Low - Medium risk applied depending on the app’s complexity functionality if answered no.
What kind of support is offered?	ORC_U33
Further Information Guidance/Context Support should be available at an equivalent quantity to the complexity of the app. This needs to be within the app or on the website, NOT an email address on the Play Store. Response Type Multiple Choice Answer Criteria Email address eTicket Live Chat Helpline/telephone number None Logic There is no disablement logic written for this question. Scoring Impact Varying risk applied depending on ESF tier of app. The higher ESF tier, the better support features are required to avoid risk.
Is there any statement within the app about the developer’s commitment to addressing problems reported to them? (e.g. timescales to respond, commitment to eradicate reported bugs and faults)	ORC_U25
Guidance Guidance/Context This question is looking to ensure that app developers are providing an SLA of sorts to their users, that they are committing to responding to and/or resolving any queries in a timely manner. Users may be more likely to reach out for support, if they know when they can expect a response. Response Type Yes/No Answer Criteria YES: If a time frame to respond is specified, it can be “we will get back to you as soon as possible”. If the website states this alongside a contact method. NO: If the statement does not give a timeframe or indication they intend to respond e.g. “we will get back to you” is not specific enough for a timescale. Logic DISABLEMENT LOGIC - Disabled if U24 is no. Scoring Impact Medium value applied if yes.
Is there a statement within the app relating to a commitment to a response in relation to healthcare queries?	ATA_UA08
Further Information Guidance/Context User support groups have service level expectations and it is important to understand how the app deals with clinical queries. For instance, will there be a timely response? Will they direct the user to a more appropriate source? Response Yes / No Assessment Criteria YES: If a time frame to respond is specified, it can be “we will get back to you as soon as possible”. YES: If the website states this alongside a contact method. Logic There is no disablement logic written for this question. Scoring Impact Medium Value applied if yes.

TECHNICAL SECURITY & STABILITY (ENHANCED REVIEW COMPONENT)

Security is one of the most challenging area for DHAF.

Overarching principles such as the Open Web Application Security Project (‘OWASP’) guidelines for mobile and web applications provide a very high level frame of reference but this doesn’t equate to a very clear set of measurable requirements..

Whilst OWASP does differentiate between different types of applications, it is a relatively crude 2 tier model and does not account for the wide range of different features and functions that digital health solutions offer

The focus is therefore switching now to a more tangible but flexible requirement that focuses on a graduated or tiered model with expected relevant security ‘credentials’ increasing as the complexity and risk of the relevant product increases.

This enables the specific features of the App and its associated security risks to be calibrated and aligned to different security characteristics /credentials.

This is however still an evolving model and the security ‘credentials’ can change in differing jurisdictions for example CISTop18, SOC2 and HITRUST in the US v Cyber Essentials and ISO 27001 in the UK.

Whilst there is a security risk associated all ‘apps’ the principle adopted in the assessment recognizes that there is a differential ‘risk profile’ for each product based on:

The Technical Architecture and related level of connectivity i.e. Attack Surface
The Data Footprint - personal, sensitive etc
The functional risk profile - i.e. simple information provision v diagnostic or treatment support
These risk profiles can be clustered into relevant risk Tiers that represents a graduated risk profile model and enables each product to be assigned to a relevant risk Tier
Each risk Tier should align to a differential set of ‘requirements’ that incrementally increase the level of expected security assurance /credentials through the Tiers

For the Security and Technical Stability assessment component these principles have been adopted in this Domain.

Tiering Guidance

Tier 1

The app is not a Medical Device, The app does not Collect or Process any Personal or Sensitive data, There is no connectivity with any other digital technology, device or system.

Tier Requirements:

Evidence that the Technology aspects of an Information Security Management System (ISMS) for the DHT Provider are in place. E.g. Cyber Essentials in the UK

Tier 2

The app is not a Medical Device, The app does not Collect or Process any Personal or Sensitive data, There is connectivity with other digital technology, device or system.

Tier Requirements:

Evidence of Security Assessment.

Use the Security Technical Compliance link here (Gray-box Penetration testing required for the Security Assessment methodology); OR

The ‘App’ is scanned dynamically and can confirm that 2 and 3 are TRUE (e.g. Kryptowire for Mobile Apps) and the Security Report can be used for Technical Security Compliance here.

Evidence that the Technology aspects of an ISMS for the ‘App’ Provider are in place and they are compliant with CIS Top 20
Tier 3

The app is not a Medical Device, The app Collects and/or Processes Personal or Sensitive data

Evidence of Security Assessment.

Use the Security Technical Compliance link here (White-Box Code Review & Gray-box Penetration testing required for the Security Assessment methodology); OR

The ‘App’ is scanned dynamically (e.g. Kryptowire for Mobile Apps) and the Security Report can be used for Technical Security Compliance here. Evidence for all aspects of an Information Security Management System (ISMS) for the ‘App’ Provider are in place and have been certified by a 3rd party assessor. E.g. ISO27001, SOC-2, etc.

Technical Stability

The Technical Stability questions are designed to capture evidence good Product and Service Management broadly covering: Robust configuration and change management processes; Responsive to user requirements and issues; Transparency in support and enhancements; and Testing approach is appropriate and robust.

Question Set

Has the developer submitted evidence for the enhanced technical security and stability assessment?	ORC_ERC_TSS01
Further Information Guidance/Context In order to under go an “enhanced review component” the developer is required to provide information and documentation which is not publicly available. in which the developer is engaged with the assessment process. Response Yes / No Answer Criteria Yes: The developer has engaged with the assessment process and provided relevant information for the Technical Security & Stability enhanced review component. No: The developer has not engaged in the assessment process at all and has not provided any relevant information for the Technical Security & Stability enhanced review component. Logic There is no disablement logic written for this question. Scoring Impact There is no scoring impact associated with this question.
Does the app connect to an internet-based API (e.g. App Developer Web Service, Social Media, Advertisements)?	ORC_ERC_OTS_C01
Further Information Guidance/Context This question is asked to help determine the needs for technical security. Response Type Yes/No Answer Criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
List the APIs the app connects to.	ORC_ERC_OTS_C02
Further Information Guidance/Context The APIs identified by this question are important to assess the appropriateness of the penetration testing carried out on the app. Response Type Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no OR ORC_ERC_OTS_C01 is no. Scoring Impact There is no scoring impact associated with this question.
Is sensitive data persisted to the mobile device?	ORC_ERC_OTS_D02
Further Information Guidance/Context This question helps determine what is the appropriate MASVS level for the platform. This is required to check the appropriateness of the Penetration test (PEN Test). Response Type Yes/No Answer Criteria Yes: If the user can input sensitive data on to the app. No: If the app does not allow the user to input sensitive data. Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
Are the source code and any configuration items for the product version controlled with all changes audited?	ORC_ERC_OTS_PSL01
Further Information Guidance/Context This is important to ensure that proper processes are followed. It also allows for changes to be checked individually as well as enabling changes to be reverted if they have been deemed to cause faults. Response Type Yes/No Answer Criteria Yes: The developer describes the people / roles that use the tools and any processes that they work to, even if these are informal. Example screenshots which demonstrate how tools are used must also be provided. Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
Provide details of any associated processes / procedures and tools that are used.	ORC_ERC_OTS_PSL02
Further Information Guidance/Context This is important to ensure that proper processes are followed. It also allows for changes to be checked individually as well as enabling changes to be reverted if they have been deemed to cause faults. This question allows the developer to describe the associated processes and procedures. Response Type Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no OR if ORC_ERC_OTS_PSL01 is no. Scoring Impact There is no scoring impact associated with this question.
Do you have the capacity to rollback to previous versions of your product?	ORC_ERC_OTS_PSL03
Further Information Guidance/Context Capacity to rollback allows the developer to resolve back to a known stable version of the product allowing any issues to be fixed within minimum disruption to the product. Response Yes/No Answer Criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
Provide details of any associated processes / procedures and tools that are used.	ORC_ERC_OTS_PSL04
Further Information Guidance/Context Capacity to rollback allows the developer to resolve back to a known stable version of the product allowing any issues to be fixed within minimum disruption to the product. This question allows the developer to describe the associated processes and procedures. Response Type Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no OR if ORC_ERC_OTS_PSL03 is no. Scoring Impact There is no scoring impact associated with this question.
Are the processes for accepting and responding to technical faults from end users appropriate?	ORC_ERC_OTS_PSL05
Further Information Guidance/Context N/A Response Type Yes/No Answer Criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
Do you provide on-line support for user queries?	ORC_ERC_OTS_PSL06
Further Information Guidance/Context The on-line support can be within the app or on the developer website but it needs to be clear you are contacting about the app or the website is for the app. Response Free Text Assessment criteria Yes: If a contact method is provided within the app, or accompanying website. Yes: If there is any way for the user to contact the developer electronically from the app. No: If the only way to contact is to provide an app store rating. No: Email addresses provided as standard on the Play Store no longer qualify. Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
Do you proactively monitor running of systems and system components to automatically identify faults and technical issues? Provide details of any associated processes / procedures and tools that are used.	ORC_ERC_OTS_PSL07
Further Information Guidance/Context This helps to identify problems faster, resulting in a stabler product as well as helping to prevent any possible breaches of information. Response Yes/No Answer Criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
Provide details of any associated processes / procedures and tools that are used.	ORC_ERC_OTS_PSL08
Further Information Guidance/Context This helps to identify problems faster, resulting in a stabler product as well as helping to prevent any possible breaches of information. This question allows the developer to describe the associated processes and procedures. Response Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no OR if ORC_ERC_OTS_PSL07 is no. Scoring Impact There is no scoring impact associated with this question.
Do you have a documented roadmap for future development of your product?	ORC_ERC_OTS_PSL09
Further Information Guidance/Context Planning for future development allows for process to be in place as well as developing the app in a structure that allows these developments to take place with minimum impact. Response Yes/No Answer Criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
Provide details of planned development, technical updates.	ORC_ERC_OTS_PSL10
Further Information Guidance/Context Planning for future development allows for process to be in place as well as developing the app in a structure that allows these developments to take place with minimum impact. This question allows the developer to describe the associated processes and procedures. Response Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no OR if ORC_ERC_OTS_PSL09 is no. Scoring Impact There is no scoring impact associated with this question.
Does the Developer provide details of how they will ensure the continued availability of their product?	ORC_ERC_OTS_PSL11
Further Information Guidance/Context Continued availability of a product is necessary because as user technology updates, the product needs to remain viable for the people that use it. Response Yes/No Answer Criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
Do you have a plan for decommissioning your product?	ORC_ERC_OTS_PSL12
Further Information Guidance/Context This is important to ensure that user data is delt with appropriately. Response Yes/No Answer Criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
Describe your processes for decommissioning your product and dealing with any identifiable data.	ORC_ERC_OTS_PSL13
Further Information Guidance/Context This is important to ensure that user data is delt with appropriately. This question allows the developer to describe the associated processes and procedures. Response Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no OR if ORC_ERC_OTS_PSL12 is no. Scoring Impact There is no scoring impact associated with this question.
Do you have a plan for dealing with any identifiable data in the event that an individual stops using your product? For example by installing or unsubscribing.	ORC_ERC_OTS_PSL14
Further Information Guidance/Context This is important to ensure that user data is delt with appropriately. Response Multiple Options Answer Criteria Yes No N/A Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no Scoring Impact There is no scoring impact associated with this question.
Provide details of any associated processes / procedures and tools that are used.	ORC_ERC_OTS_PSL15
Further Information Guidance/Context This is important to ensure that user data is delt with appropriately. This question allows the developer to describe the associated processes and procedures. Response Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no OR if ORC_ERC_OTS_PSL14 is no or N/A. Scoring Impact There is no scoring impact associated with this question.
Does the App Developer have robust Disaster Recovery (DR)/ back-up regimes in place?	ATA_PSL01
Further Information Guidance/Context Evidence should include documented processes and procedures. Response Yes/No Answer Criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
When were these last tested?	ATA_PSL02
Further Information Guidance/Context The question aims to discover when the DR / back up regimes was last tested. Good practice is for these to be tested at least annually. Response Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no OR if there is no robust DR / back-up regimes in place. Scoring Impact There is no scoring impact associated with this question.
Business Resilience: Does the app developer have a Business Continuity Plan (BCP) in place?	ATA_PSL03
Further Information Guidance/Context Evidence should include documented processes and procedures. Response Yes/No Assessment Criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
When were these last tested?	ATA_PSL04
Further Information Guidance/Context The question aims to discover when the BCP was last tested. Good practice is for these to be tested at least annually. Response Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no OR if there is no business continuity plan in place. Scoring Impact There is no scoring impact associated with this question.
Does the organization follow any formal testing standards?	ORC_ERC_OTS_PSL16
Further Information Guidance/Context Evidence should include documented processes and procedures if they exist, and screenshots of any tools that are used. Evidence of formal certification can also be provided. Response Yes/No Assessment criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no Scoring Impact There is no scoring impact associated with this question.
Provide details of any associated processes / procedures and tools that are used.	ORC_ERC_OTS_PSL17
Further Information Guidance/Context Evidence should include documented processes and procedures if they exist, and screenshots of any tools that are used. Evidence of formal certification can also be provided. Response Free Text Assessment criteria N/A Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no OR if no formal testing standards are followed. Scoring Impact There is no scoring impact associated with this question.
Which of these types of testing do you carry out?	ORC_ERC_OTS_PSL18
Further Information Guidance/Context Formal test plans, checklists and screenshots of tools can be provided as evidence. Response Multiple Choice Answer Criteria Unit testing Regression End-to-end User Acceptance A/B PEN/Vulnerability Testing across devices Load / Performance Security Other non-functional tests Other testing None Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
For unit testing please describe the people / roles that are involved, the processes that they work to even if they are informal.	ORC_ERC_OTS_PSL19
Further Information Guidance/Context This question aims to identify what personnel and processes are undertaken to perform this testing. Response Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no OR if the developer does not carry out Unit testsing. Scoring Impact There is no scoring impact associated with this question.
For Regression testing please describe the people / roles that are involved, the processes that they work to even if they are informal.	ORC_ERC_OTS_PSL20
Further Information Guidance/Context This question aims to identify what personnel and processes are undertaken to perform this testing. Response Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no OR if the developer does not carry out Regression testing. Scoring Impact There is no scoring impact associated with this question.
For End-to-end / Integration please describe the people / roles that are involved, the processes that they work to even if they are informal.	ORC_ERC_OTS_PSL21
Further Information Guidance/Context This question aims to identify what personnel and processes are undertaken to perform this testing. Response Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no OR if the developer does not carry out End-to-end/Integration testing. Scoring Impact There is no scoring impact associated with this question.
For User Acceptance please describe the people / roles that are involved, the processes that they work to even if they are informal.	ORC_ERC_OTS_PSL22
Further Information Guidance/Context This question aims to identify what personnel and processes are undertaken to perform this testing. Response Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no OR if the developer does not carry out User Acceptance testing. Scoring Impact There is no scoring impact associated with this question.
For A/B please describe the people / roles that are involved, the processes that they work to even if they are informal.	ORC_ERC_OTS_PSL23
Further Information Guidance/Context This question aims to identify what personnel and processes are undertaken to perform this testing. Response Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no OR if the developer does not carry out A/B testing. Scoring Impact There is no scoring impact associated with this question.
For PEN / Vulnerability please describe the people / roles that are involved, the processes that they work to even if they are informal.	ORC_ERC_OTS_PSL24
Further Information Guidance/Context This question aims to identify what personnel and processes are undertaken to perform this testing. Response Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no OR if the developer does not carry out PEN / Vulnerability tests. Scoring Impact There is no scoring impact associated with this question.
For Testing across devices please describe the people / roles that are involved, the processes that they work to even if they are informal.	ORC_ERC_OTS_PSL25
Further Information Guidance/Context This question aims to identify what personnel and processes are undertaken to perform this testing. Response Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no OR if the developer does not carry out Testing cross devices. Scoring Impact There is no scoring impact associated with this question.
For Load / Performance please describe the people / roles that are involved, the processes that they work to even if they are informal.	ORC_ERC_OTS_PSL26
Further Information Guidance/Context This question aims to identify what personnel and processes are undertaken to perform this testing. Response Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no OR if the developer does not carry out Load / Performance tests. Scoring Impact There is no scoring impact associated with this question.
For Security please describe the people / roles that are involved, the processes that they work to even if they are informal.	ORC_ERC_OTS_PSL27
Further Information Guidance/Context This question aims to identify what personnel and processes are undertaken to perform this testing. Response Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no OR if the developer does not carry out Security tests. Scoring Impact There is no scoring impact associated with this question.
For Other non-functional tests please describe the people / roles that are involved, the processes that they work to even if they are informal.	ORC_ERC_OTS_PSL28
Further Information Guidance/Context This question aims to identify what personnel and processes are undertaken to perform this testing. Response Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no OR if the developer does not carry out Other Non-functional tests. Scoring Impact There is no scoring impact associated with this question.
For Other testing please describe the people / roles that are involved, the processes that they work to even if they are informal.	ORC_ERC_OTS_PSL29
Further Information Guidance/Context This question aims to identify what personnel and processes are undertaken to perform this testing. Response Free Text Answer Criteria N/A Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no OR if the developer does not carry out Other testing. Scoring Impact There is no scoring impact associated with this question.

Technical Security

Whilst there is a security risk associated with all Digital Health Technologies, the principle adopted in assessment reflects the differential ‘risk profile’ for each product. Risk Tiers can be based on:

Technical Complexity: Is there a high degree of digital connectivity, functionality and potential “Attack Surface”?
Data: Is the data personal and/or sensitive? Where and how is it stored and transmitted?
Functional Complexity: Is the product providing information v diagnostic or treatment support? Are there data driven calculations and algorithms?

Question Set

Is the application a native application for a mobile device?	ORC_ERC_SEC01
Further Information Guidance/Context This helps guide what OWASP/ASVS level is required surrounding the PEN test. Response Yes/No Answer criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
Is the application a web application?	ORC_ERC_SEC02
Further Information Guidance/Context This helps guide what OWASP/ASVS level is required surrounding the PEN test. Response Yes/No Answer criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no Scoring Impact There is no scoring impact associated with this question.
What OWASP level is the app?	ORC_ERC_SEC07
Further Information Guidance/Context This question aims to identify what OWASP level the app should be by referring to the answers of the previous questions. Response Multiple Options Assessment Criteria MASVS = 2: If the platform is a mobile app and it does accesses, processes or stores personal and/or sensitive data MASVS = 2 + R: If the platform is a mobile app and sensitive data persists on the mobile device MASVS = 1: If the platform is a mobile app and sensitive data does not persist on the mobile device OR If the platform is a mobile app and it does not access, process or store personal and/or sensitive data ASVS = 2: If the platform is a web app and it accesses, processes or stores personal/sensitive data ASVS = 1: If the platform is a web app and it does not access, process or store personal/sensitive data Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
Has a Security Assessment been undertaken by an external third-party?	ATA_SEC05
Further Information Guidance/Context This provides assurance that the PEN test will have been scoped appropriately and the methodology will also be appropriate. Response Yes/No Answer criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
Does the scope of the report cover the full Technical Architecture of Application?	ORC_ERC_SEC16
Further Information Guidance/Context All platforms and technical components should be in scope of the Security Assessment. Response Yes/No Answer criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
Has an industry-standard been used for the risk model in the associated PEN /Vulnerability testing?	ORC_ERC_SEC17
Further Information Guidance/Context This provides assurance that the PEN test has been executed professionally e.g. Common Vulnerability Scoring System (CVSS). Response Yes/No Answer criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
Have all 'High' and ‘Medium’ Risks / Issues identified been mitigated and resolved; and can this be demonstrated through retesting within six weeks from the original PEN / Vulnerability testing?	ORC_ERC_SEC18
Further Information Guidance/Context Evidence should include the full version of the original PEN test report and any retest. Response Yes/No Answer criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
Has the Code-Level Security Assessment been undertaken against the correct OWASP Level?	ORC_ERC_SEC19
Further Information Guidance/Context This level will be detailed in the Security Assessment report and any associated PEN testing report. Response Yes/No Answer criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
Is the methodology for the Security Review proportional to the attack surface and risk of the Application?	ORC_ERC_SEC20
Further Information Guidance/Context The scope and methodology should be proportional to the associated risk. Response Yes/No Answer criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
Does the organization have CIS Top 20 Compliance?	ATA_SEC01
Further Information Guidance/Context Evidence should include formal certification certificates. Response Yes/No Answer criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
Is the application compliant with the National Institute for Standards and Technology (NIST) Cybersecurity Framework?	ATA_SEC06
Further Information Guidance/Context Evidence should include formal certification certificates. Response Yes/No Answer criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
Does the organization have SOC-2 Certification?	ATA_SEC01
Further Information Guidance/Context Evidence should include formal certification certificates. Response Yes/No Answer criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
Does the organization have ISO27001:2013 accreditation?	ORC_ERC_SEC_ORG1
Further Information Guidance/Context Evidence should include formal certification certificates and the associated Statement of Applicability should include product development. Response Yes/No Answer criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
Does the organization have ISO 13485 Certification?	ATA_SEC03
Further Information Guidance/Context Evidence should include formal certification certificates. Response Yes/No Answer criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
Does the organization have ISO 14971 Certification?	ATA_SEC04
Further Information Guidance/Context Evidence should include formal certification certificates. Response Yes/No Answer criteria Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.
Please confirm whether the product has passed the required criteria for the Technical Security & Stability section of the assessment?	ATA_TSS01
Further Information Guidance/Context If the app fulfils the above criteria they will pass. There are some instances where apps may fail to meet one criterion and then this can be considered on a case by case basis with a subject matter expert as to whether they will pass or fail. Response Multiple Options Answer criteria Pass: If they fulfil all the above criteria. Fail: If they cannot meet the above criteria. Logic DISABLEMENT LOGIC - Disabled if ORC_ERC_TSS01 is no. Scoring Impact There is no scoring impact associated with this question.

Are users clearly informed of the use of cookies when first landing on the developers site/app?	ORC_D99
Further Information Guidance/Context When reviewing a native (including “hybrid”) app, being informed of the website using cookies, while using the browser, does not answer this question as yes. Reference to “site” in the question is regarding a review of a web app. Response Type Yes/No Answer Criteria Yes (web app): If when first landing on the website a banner/pop up appears asking users to accept cookies. Yes (Native Apps): If when first accessing the app, or at the point at which the app attempts to use cookies, the user is clearly informed of the intended use of cookies. Logic DISABLED LOGIC - Disabled if the app is not collecting cookies, D01 is answered No or DT10 does not contain “Cookies/Web Beacons” Scoring Impact Low risk if answered No + Multiplier based on the nature of the data.
Are user's required to confirm their acceptance of the developer's use of cookies, when initially informed of the use?	ORC_D100
Further Information Guidance/Context It is required that developers gain consent from visitors to the site in order to store or retrieve any information on a computer, smartphone or tablet using cookies. Response Type Yes/No Answer Criteria Yes (web app): If when first landing on the website a banner/pop up appears asking users to accept cookies. Yes (Native Apps): If, when users are informed of the use of cookies, they are required to provide a clear confirmation of their acceptance of the use of cookies. Logic DISABLED LOGIC - Disabled if the app is not collecting cookies, D01 is answered No or DT10 does not contain “Cookies/Web Beacons” Scoring Impact Low risk if answered No + Multiplier based on the nature of the data.
Does the developer address their use of cookies and collected data in their Privacy Policy, or in a separate Cookie Policy?	ATA_DP24
Further information Guidance/Context A paragraph explaining the products use of cookies is sufficient for this question. Response Yes / No Assessment Criteria Yes: If there is a designated paragraph or section within the privacy policy which explains the developer’s use of cookies. Yes: If there is a whole separate cookie policy for the app. No: If the cookie policy only addresses cookies for an associated website and not the app itself. No: If there is no cookie policy or paragraph made available to the user. Logic DISABLED LOGIC - Disabled if DT11 is no. Scoring Impact Low Risk applied if no.
Are users made aware of the use of strictly necessary cookies?	ORC_ERC_EDC_CK04
Further information Guidance/Context Users should be informed of the use of strictly necessary cookies, even if they have no way of restricting the use of these. Response Yes / No Assessment Criteria Yes: If there is a statement in the privacy policy. Yes: If there is a pop up explaining the use of strictly necessary cookies. No: If there is no statement anywhere explaining the user of strictly necessary cookies. Logic DISABLED LOGIC - Disabled if DT11 is no. Scoring Impact Low Risk applied if no + multiplier depending on nature of the data that’s collected.
Is user consent obtained for the use of non strictly necessary cookies?	ORC_ERC_EDC_CK05
Further information Guidance/Context Under the e-Privacy Directive 2002, manufacturers should obtain separate consent for their use of non-strictly necessary cookies. Response Yes / No Assessment Criteria Yes: If there is a pop up on the app which asks the user to consent to the use of non strictly necessary cookies. Yes: If user is asked to provide their consent to non strictly necessary cookies upon sign up. No: If there is no form of consent obtained for non strictly necessary cookies. Logic DISABLED LOGIC - Disabled if DT11 is no. Scoring Impact Low Value applied if yes.
Are users informed of how they can easily opt out of the use of cookies?	ORC_ERC_EDC_CK07
Further information Guidance/Context Under the e-Privacy Directive 2002, manufacturers are required to ensure it is as easy for user’s to opt out of their use of cookies as it was for the user to originally opt in. Response Yes / No Assessment Criteria Yes: If there’s a statement within the privacy policy or on the app about how to opt out of the use of cookies. No: If the user has not been told how to opt out of the use of cookies. Logic DISABLED LOGIC - Disabled if DT11 is no. Scoring Impact Low Risk applied if no + multiplier based on the nature of the data collected.
Is the product aimed at children or likely to be used by children?	ORC_ERC_EDC_COP01
Further information Guidance/Context The question aims to identify if the app is targeted at children, as additional security measures should be in place if a minor’s data is being processed. Response Yes / No Assessment Criteria Yes: The app has stated it’s for child use. No: The policy is quite clear that the app is aimed at people over 18 or they won’t take data from an under 18. Logic DISABLED LOGIC - Disabled if the app doesn’t collect personal data OR if the app doesn’t share any data OR if the data is only shareable through direct actions by the user. Scoring Impact None.
Is the app ‘particularly likely’ to be used by children, even if they are not the primary market for the app?	ORC_D44
Further Information Guidance/Context The question aims to identify if the app may collect data from children as additional security measures should be in place if a minor’s data is being processed. Response Type Yes/No Answer Criteria Yes: The app has content which may be appealing to children, but doesn’t specify an age range, OR the app is intended to be used by children. No: If policy states they won’t collect data from under 13s, GDPR still allows for the 13-16 age range to provide consent independently No: The policy is quite clear that the app is aimed at people over 18 or they won’t take data from an under 18 AND the app does not present any particular content or features that would encourage a minor to attempt to access and use the app. Logic DISABLED LOGIC - Disabled if D01 is no, Or DS06 is answered yes, Or if DT13 is answered none personal. Scoring Impact None
Are users informed of how they can report, to the developer, any knowledge of a child accessing the app and providing personal data, without parental consent?	ORC_DO01
Further Information Guidance/Context This question aims to encourage developer’s to provide the opportunity for people to contact them if they believe a child’s data may have been processed or collected incorrectly or without parental consent. Response Type Yes/No Answer Criteria Yes: If there is a statement that specifically details what a user should do to inform the developer. Yes: If the developer specifically states that if they become aware of a minor/child providing personal data, then they will delete this data within a set period of time. Yes: If the developer explains the app is offering online preventive or counselling services to children and therefore does not obtain parental consent as they are legally obliged to not do so. No: If the policy does not provide any details on how the developer and/or user should respond when they become aware of a minor/child providing personal data. Logic DISABLED LOGIC - Disabled if D01 is no, Or DS06 is answered yes, Or if DT13 is answered none personal. Scoring Impact High, Medium or Low risk applied depending on if D44 is answered yes, if the app is designed for child, Pre-teen or teen with a multiplier applied based on the level of data collected (Non-personal, Personal, Sensitive)
Has a process been designed and put in place that allows children to easily access, understand and exercise their own data protection rights?	ORC_ERC_EDC_COP02
Further Information Guidance/Context User rights should be written in a form that the child can understand them, so they know what they are agreeing to and steps they can take to exercise their rights, this is applied to users under the age of 13 too. Response Type Yes/No Answer Criteria Logic DISABLED LOGIC - Disabled if COP01 is no. Scoring Impact Low risk applied is no + multiplier based on the nature of the data.
Where consent was the legal basis for processing data was consent, at the time the individual was a child, then requests for the erasure of data are complied with, whenever possible?	ORC_ERC_EDC_COP04
Further Information Guidance/Context Response Type Yes/No Answer Criteria Logic DISABLED LOGIC - Disabled if COP01 is no. Scoring Impact Low value applied if yes.
Have children been consulted when designing this processing practice?	ORC_ERC_EDC_COP05
Further Information Guidance/Context This question aims to discover if children were involved in the process of making their user rights easily accessible, understandable and exercisable. Response Type Yes/No Answer Criteria Logic DISABLED LOGIC - Disabled if COP01 is no. Scoring Impact Low value applied if yes.
Has the privacy policy been written in plain, age appropriate language?	ORC_ERC_EDC_COP06
Further Information Guidance/Context This question aims to discover if the privacy policy has been written in a language which is plain and understandable to the app’s target audience. Response Type Yes/No Answer Criteria Logic DISABLED LOGIC - Disabled if COP01 is no. Scoring Impact Low risk value applied if no + multiplier based on nature of the data.
Is consent sought from a responsible parent /guardian?	ORC_ERC_EDC_COP07
Further Information Guidance/Context This question aims to identify consent from a child’s data use is sought from a responsible parent/guardian. Response Type Yes/No Answer Criteria Logic DISABLED LOGIC - Disabled if COP01 is no. Scoring Impact None.
Does the developer ensure they do not seek parental/guardian consent when providing online preventive or counselling services to children?	ORC_ERC_EDC_COP08
Further Information Guidance/Context In order to help protect and safeguard children when providing preventive or counselling services, a developer must ensure that the parent/guardian is not aware of the child’s access to the service, as this could result in exposing the child to harm. Response Type Yes/No Answer Criteria Logic DISABLED LOGIC - Disabled if COP01 is no OR COP07 is yes. Scoring Impact Low risk applied if no + multiplier based on nature of data collected.
Does the policy specify that the developer will reobtain parental consent, should the information collected materially changes, the purpose upon which information is processed changes, or the information is to be offered to new/different third parties?	ATA_DP15
Further Information Guidance/Context Similar to the importance of reobtaining consent from any user, when the purpose of processing changes (or a new purpose arises), where parental consent has been obtained, the consent of the parent should be reobtained. Response Type Yes/No Answer Criteria Logic DISABLED LOGIC - Disabled if COP01 is no OR COP08 is yes. Scoring Impact Low risk applied if no + multiplier based on nature of data collected.
Does the developer ensure that parents are able to separately consent to their own internal use of the child’s personal information, without having to consent to the disclosure of personal information to third parties?	ATA_DP16
Further Information Guidance/Context It is important that separate consent is gained for sharing data with third parties for processing and the collection of data by the developer for their own internal processing. Response Type Yes/No Answer Criteria Logic DISABLED LOGIC - Disabled if COP01 is no OR COP08 is yes. Scoring Impact Low risk applied if no + multiplier based on nature of data collected.
Are parents given the option to review the personal information collected from their children?	ATA_DP17
Further Information Guidance/Context Since children need a responsible parent or guardian to consent to how their data is being used, the parent would also be entitled to the same rights as any user who provides their data. Response Type Yes/No Answer Criteria Logic DISABLED LOGIC - Disabled if COP01 is no OR COP08 is yes. Scoring Impact Low risk applied if no + multiplier based on nature of data collected.
Does the developer have a process for verifying the identity of the requester, before responding to a request?	ATA_DP18
Further Information Guidance/Context It’s important that the person who is requesting to view the child’s data is who they say they are as well as the responsible parent/guardian for that child else data could get in the wrong hands. Response Type Yes/No Answer Criteria Logic DISABLED LOGIC - Disabled if COP01 is no. Scoring Impact Medium value applied is yes.
Are parents given the option to revoke consent for the collection and processing of their children’s personal information?	ATA_DP19
Further Information Guidance/Context Since children need a responsible parent or guardian to consent to how their data is being used, the parent would also be entitled to the same rights as any user who provides their data. Response Type Yes/No Answer Criteria Logic DISABLED LOGIC - Disabled if COP01 is no. Scoring Impact Low risk applied if no + multiplier based on nature of data collected.
Are parents given the option to request that the information collected from their children be deleted?	ATA_DP20
Further Information Guidance/Context Since children need a responsible parent or guardian to consent to how their data is being used, the parent would also be entitled to the same rights as any user who provides their data. Response Type Yes/No Answer Criteria Logic DISABLED LOGIC - Disabled if COP01 is no. Scoring Impact Low risk applied if no + multiplier based on nature of data collected.
Are there two separate versions of privacy policies, one aimed at the child and the other at the responsible parent/guardian?	ORC_ERC_EDC_COP09
Further Information Guidance/Context A privacy policy be written in a language that the user can understand it therefore it the user is a child, there should be an appropriate policy in place which they can understand. Additionally, if a responsible parent/guardian is required to provide consent on behalf of their child, an appropriate policy should be in place which is aimed at this user group. Response Type Yes/No Answer Criteria No: If only one policy is available. Logic DISABLED LOGIC - Disabled if COP01 is no. Scoring Impact Low value applied if yes.
When marketing the product outside of their country of residence, has the developer taken into consideration other jurisdictional laws regarding children’s privacy (eg. age restrictions)?	ORC_ERC_EDC_COP10
Further Information Guidance/Context If the developer intends for the app to be used in other countries, it is important they take into account how old a ‘child’ is defined in that country as this can differ between nations. Response Type Yes/No Answer Criteria Logic DISABLED LOGIC - Disabled if COP01 is no. Scoring Impact Low risk applied if no + multiplier based on nature of data collected.
Does the policy specify the types of personal data that will be collected from children?	ATA_DP21
Further Information Guidance/Context Based on app and privacy policy, see whether it specifies what personal is collected from children. Response Type Yes/No Answer Criteria Logic DISABLED LOGIC - Disabled if COP01 is no. Scoring Impact Low risk applied if no + multiplier based on nature of data collected.
Does the policy specify how the developer will use the personal data collected from children?	ATA_DP22
Further Information Guidance/Context Based on app and privacy policy, see whether the policy explains how personal data will be used. Response Type Yes/No Answer Criteria Logic DISABLED LOGIC - Disabled if COP01 is no. Scoring Impact Low risk applied if no + multiplier based on nature of data collected.
Does the policy specify whether such personal data will be shared with advertisers or other third parties?	ATA_DP23
Further Information Guidance/Context It is important that the policy is completely transparent about where and with who the data is shared. This may cause the responsible parent/guardian to reconsider whether the child should be using the app or not. Response Type Yes/No Answer Criteria Logic DISABLED LOGIC - Disabled if COP01 is no. Scoring Impact Low risk applied if no + multiplier based on nature of data collected.
Is the user made aware that by following links to third party websites, the developer’s policies no longer apply, and that the user should make themselves aware of the third party’s policies?	ORC_D91
Further Information Guidance/Context Developers should make users aware that they should make themselves aware of third party policies as the developers privcay policy no longer applies. This may also be found in the Terms & Conditions. Response Type Yes/No Answer Criteria Yes: The policy mentions that the developer’s policy doesn’t extend to third parties and users are advised to make themselves aware of the privacy policies of any third party site/platform that they visit through the app. Yes: Users are provided links to relevant third party privacy policies. Logic DISABLED LOGIC - Disabled if D39a And D39b have not been answered Yes. Scoring Impact Medium value score applied if D91 has been answered Yes.
Is the user informed of how they can make further enquiries about the company’s privacy policy?	ORC_D92
Further Information Guidance/Context This question is looking to capture if the user has the ability to contact developers if they have questions around the privacy policy and their processes. Response Type Yes/No Answer Criteria Yes: The user is informed of/given a method of contact for any queries regarding the policy. Contact method must be in the policy, normally found toward the end of the policy. No: If it says “contact us” but no contact method is given in the policy. No: If there is no clear statement that directs the user to contact information for the purpose of enquiring about the company's privacy policy/practices. Logic DISABLED LOGIC - Disabled if D39a And D39b have not been answered Yes. Scoring Impact Medium value score applied if D92 has been answered Yes.
Does the app allow the user to set their preferences for sharing the app data with or from other apps (e.g. Facebook / Instagram /Fitbit etc)?	ORC_D06
Further Information Guidance/Context This question looks to capture any sharing preferences includes sharing with third parties, not necessarily other apps. Response Type Yes/No Answer Criteria Yes: If the app allows user controls over data sharing with individual apps/platforms. I.e. gives the choice to turn on/off sharing with google fit, Instagram, Fitbit etc. Yes: If it allows sign up through Facebook/Google+ or separately with an email address, as this gives the option to sign up with or without sharing with/from other apps/platforms. No: If the app doesn’t ask permission to share to other apps, but does so automatically. Even if this is based on user agreement to privacy policies/T’s & C’s. No: If ONLY sign up with Facebook/Google+ is allowed, i.e. you have no choice but to do so. Logic DISABLED LOGIC - Disabled if DS03 is answered No. Scoring Impact Medium value score applied if D06 has been answered Yes.
Is there functionality within the app to allow the user to set their preferences for sharing app data with others users (physicians, caregiver, family, friends, buddies)?	ORC_D27
Further Information Guidance/Context This question is looking to identify if users can choose if information is shared with other users. Added control over users own data is beneficial for maintaining privacy where appropriate. Response Type Yes/No Answer Criteria Yes: Can choose WHAT is being shared with WHO on the app. Yes: If the data is only ever shared with other users through manual user intervention. E.g. users choose to post on a forum/news feed. No: The app gives no control over who sees what. E.g. an open forum/send to all clinicians. Logic DISABLED LOGIC - Disabled if DS03is answered No. Scoring Impact Medium value score applied if DS03 has been answered Yes.
Is it strictly necessary for anyone to easily access the personal information that persists on the device? e.g. to access health info during an emergency.	ORC_DO02
Further Information Guidance/Context This question is specific to the app on the device it has been downloaded onto. e.g Is there a need to access information stored on the app during an emergency. Response Type Yes/No Answer Criteria Yes: If the intended purpose of the app is to provide information to those providing emergency response, in the event that the individual concerned is unable to communicate with the responder. No: If access is for any other reason, including if access is remote for clinicians to monitor patients. Logic DISABLED LOGIC - Disabled if DS03 is answered No. Scoring Impact Medium value score applied if DS03 has been answered Yes.
Are users provided options to introduce additional security measures to protect their data on the app? eg. set additional pass codes for access to the app, after accessing the device is unlocked.	ORC_DO03
Further Information Guidance/Context Data contained within an app may be private to an individual. Adding security features to the app itself reassures users. Response Type Yes/No Answer Criteria Yes: App allows a pass code to be set, or use device security/unlock mechanisms a second time to access the app. Yes: If there is the option to choose who can see information contained within the user profile/set privacy controls on you account. No: If the user can set data sharing controls, such as choosing apps to share data with. Logic DISABLED LOGIC - Disabled if DO02 has been answered Yes. Scoring Impact High value score applied if DO03 has been answered Yes.
Does the app use a sign up/sign in verification/authentication model?	ORC_DO04
Further Information Guidance/Context This question is looking to identify if the users access/identify is verified in any way. This is important to ensure the person creating the account is who they say they are and has access to the related accounts e.g. Email address. Response Type Yes/No Answer Criteria Yes: If there are any forms of user authentication being used. No: If the developer does not have any way, beyond signing in, by which they verify that the person creating/accessing an account, is the person that they claim to be/the owner of the account Logic DISABLED LOGIC - Disabled if DT14 has been answered no. Scoring Impact High value score applied if DO04 has been answered Yes.
What type of model is being used?	ORC_DO05
Further Information Guidance/Context N/A Response Type Multiple Option Answer Criteria None One-step email authentication - if already signed up, check by resetting password, if email link is sent to reset it is this one. Other one-step authentication - e.g. Biometric access, pin number HCP Granted Access/Invite - A referral code needed to access the app which comes from the HCP. Admin Granted Access/Invite - Healthcare provider granting access to each individual HCP/user. SMS authentication - code sent to phone confirming it is you signing in Two-step authentication - Use of a separate authenticator app or a code sent to phone/email whenever you sign into the app which needs to be confirmed. The app uses a second authentication step after the user has clicked an email verification link when signing up e.g. requests a mobile number and sends a verification code by text. Multi-step authentication - any more than 2 steps Qualification/HCP Registration Check - Being able to register as a clinician and having your credentials checked before being accepted as a HCP to provide information to patients. Identification Check (Eg. drivers licence, passport) - scan/take photos of ID for sign up purposes or ID verification e.g. NHS app when signing up or for a clinician to verify the person they are speaking to is the patient they are supposed to be dealing with. Logic DISABLED LOGIC - Disabled if DT14 has been answered no. Scoring Impact None

Digital Health Assessment Framework V1.0

INRODUCTION

Assessment Domains

Value and Risk Points

SCENE SETTERS

App Characteristics

Data - Data Types, Data Collection and Data Sharing

Algorithm/AI

Information

Clinical Decision Support - Pre-Diagnosis, Diagnosis and Treatment Support

Monitoring

Online Consultations

Administrative Services

Pharmacy

Reminders/Notifications

External Device

Forums and Contacts

Goal Setting

Customization

Business Model

Benefits

DATA & PRIVACY

Need for HIPAA Compliance

Privacy Policy

Data use

Data Storage and Transit/Transfer

Data Standards and Management

GDPR/HIPAA/Jurisdictional Principles

Other Data Questions

CLINICAL ASSURANCE & SAFETY

Professional Assurance

Evidence of Effectiveness

Behavioral Change

Professional Backing

ORCHA Adapted ESF Compliance

Medical Devices

Safety/Risk Management

USABILITY AND ACCESSIBILITY

Design and Development

Accessibility

Usability

Support

TECHNICAL SECURITY & STABILITY (ENHANCED REVIEW COMPONENT)

Technical Security