Nordic Digital Health Evaluation Criteria
Watch for changes
Introduction
The Nordic Digital Health & Medication Platform project aims to establish a system for healthcare providers to evaluate and identify trusted digital health technologies within healthcare and preventive care. These digital health technologies will be evaluated by the Nordic Digital Health Evaluation Criteria (NordDEC). Below you will find the scoring and questions that make up the criteria.
Value and Risk Points
The scoring is made up of Value earning points and Risk earning points.
Each scoring question has either a Risk implication or a Value implication
The quantum of the Risk or Value implication is decided by the relevant tariff which range from small, medium, high or exceptionally high in the Risk area and small, medium or high in the Value area.
The following table sets out the actual numeric value of each Tariff:
Tariff |
Risk |
Value |
---|---|---|
Small |
10 |
5 |
Medium |
20 |
10 |
High |
40 |
20 |
Exceptionally High |
80 |
- |
In addition to the base tariff, some risk and value related questions attract a multiplier that will increase the relevant tariff based on certain related app characteristics.
Maximum risk can be applied based on responses for certain questions. Maximum risk is applied to a whole section (i.e Data), rather than an individual question. It is the sum of all the risk points that could be applied if were not for the questions being disabled by earlier responses.
SCENE SETTERS
The NordDEC begins with a series of questions designed to assess core purpose and functionality of digital health products in the form of native apps or web apps (“apps”). For the purpose of the NordDEC a digital health app is defined as “a digitally delivered product that is aimed at supporting in some way general health or specific conditions”. The questions look to capture the target audience, the type of data the app collects and the apps primary functions and features. None of the scene setter questions are intended to have any scoring or risk implications and are purely to decide on the line of enquiry further in the evaluation.
Every question within scene setters does not have a scoring value.
App Characteristics
Question |
Question Reference Source |
Is the App health focused? |
ORC_SS01 |
---|---|
Data - Data Types, Data Collection and Data Sharing
Question |
Question Reference Source |
Does the App collect data? |
ORC_D01 |
What type of data is collected by the App? |
ORC_DT10 |
---|---|
What Permissions does the app request? |
ORC_ERC_OTS_P01 |
Are users required/able to sign up/register to use the service? |
ORC_DT14 |
Is data collected through cookies? |
ORC_DT11 |
What type of cookies are used? |
ORC_DT12 |
Is the data (cookie and/or none cookie) collected: |
ORC_DT13 |
How is non-cookie data collected? |
ORC_DC01 |
What other apps is the App connected to? |
ORC_DC02 |
What device(s) does the App connect to? |
ORC_DC03 |
Can the user prevent cookie data being collected and still use the App? |
ORC_DS01 |
Does the disabling of cookies impact the use of the App in any way? |
ORC_DS02 |
Can/is data shared? (excluding cookies) |
ORC_DS03 |
Can data be shared through a direct, manual action by the user? {e.g. by sending data via email or manually choosing to post/share something within the app etc} |
ORC_DS04 |
How is the user able to manually share their data? |
ORC_DS05 |
Is data ONLY shareable through a direct, manual action by the user? (excluding cookies) |
ORC_DS06 |
Can the user control any automatic data sharing, through setting individual sharing preferences in the app? (excluding cookies) |
ORC_DS07 |
Where/With who can the user share data automatically by manually setting sharing preferences in the app? |
ORC_DS08 |
Is any data (excluding cookie data) shared automatically as soon as the App is accessed – based only on agreement to relevant Terms of Use or Privacy Policy? |
ORC_DS09 |
Where/With who is data automatically shared - based only on user agreement to the developer’s Privacy Policy and/or Terms of Use? |
ORC_DS10 |
What data is automatically shared with the developer? |
ORC_DS12 |
What data is automatically shared with physicians / healthcare professionals? |
ORC_DS13 |
What data is automatically shared with other users? |
ORC_DS14 |
What data is automatically shared with third parties? |
ORC_DS15 |
What data is automatically shared with other devices? |
ORC_DS16 |
Algorithm/AI
Question |
Question Reference Source |
Does the app contain algorithms? |
ORC_AI01 |
---|---|
How does the app use the algorithm? |
ORC_AI02 |
Does the app appear to use AI? |
ORC_AI03 |
What AI technique is used in the app? |
ORC_AI04 |
Is the AI monitored/ maintained? |
ORC_AI05 |
Information
Question |
Question Reference Source |
Is the app designed to provide information or guidance? |
ORC_I01 |
---|---|
Does the app provide information that is personalised to an end user’s specific circumstances? |
ORC_I02 |
Does the app provide users with information regarding where they are able to find local or suitable support services? |
ORC_F08 |
Does the app provide environmental data not specific to the patient? |
ORC_F03 |
Does the App provide information, resources or activities to the public, patients or physicians, either about a specific condition or general health and lifestyle? |
ORC_EF07 |
Clinical Decision Support - Pre-Diagnosis, Diagnosis and Treatment Support
Question |
Question Reference Source |
Is the data the app collects, automatically assessed, for the purposes of evaluating: risk; or providing diagnostic support? |
ORC_PD01 |
---|---|
Does the app diagnose a specific condition? |
ORC_DG02 |
Does the app provide an assessment (of the risk) to an individual - based on data input or collected by the app - of: Contracting or Suffering a healthcare condition The impact on their lifestyle and health indicators No Risk Assessment provided |
ORC_DG01 |
Does the app provide an assessment (of the risk) to a healthcare professional - based on data input or collected by the app - of: Contracting or Suffering a healthcare condition The impact on their lifestyle and health indicators No Risk Assessment provided |
ORC_DG03 |
Does the app provide the option for further assessment or analysis by a healthcare professional? |
ORC_DG04 |
Is the app/does the app include a Symptom Checker? |
ORC_DG05 |
Does the app indicate likelihood of a match for the listed conditions? |
ORC_DG06 |
Can users filter results to display by highest risk / likelihood / severity? |
ORC_DG07 |
Does the app provide treatment recommendations for the listed conditions? |
ORC_DG08 |
Does the app only signpost the user to suitable care or recommend seeking further advice? (eg. Go to ER, book an appointment with your family physician, call 911) |
ORC_DG09 |
Does the app contain a clinical calculator? |
ORC_TS01 |
What type of clinical calculator does the app contain? |
ORC_TS02 |
Is the app intended to be (or does the developer claim it can be) used for the prevention of disease? |
ORC_MD01 |
How does the app prevent disease? |
ORC_TS04 |
Does the app provide treatment of a condition? |
ORC_TS05 |
What treatment does the app provide? |
ORC_TS06 |
Does the app guide the treatment of a condition? |
ORC_TS07 |
How does the app guide the treatment of the condition? |
ORC_TS08 |
Who does the app provide the treatment guidance to? |
ORC_TS09 |
Is the treatment provided independently of a healthcare professional? |
ORC_TS10 |
Does the app support healthcare professionals’ decisions about treatments? |
ORC_TS03 |
Does the app follow the path of a procedure/treatment without making any decisions? |
ORC_TS12 |
Does a healthcare professional make the final decision regarding treatment based on advice and/or options displayed? |
ORC_TS13 |
Does the app automate the treatment pathway for an individual patient? |
ORC_TS14 |
Is the app intended to be (or does the developer claim it can be) used as a physical intervention to reduce the symptoms or severity of a disease, injury or, physical or mental impairment? |
ORC_TS15 |
Is the app intended to (or does the developer claim it can be used to) compensate an injury or, physical or mental impairment? |
ORC_MD07 |
Does the app predict the fertile window? |
ORC_CC01 |
Does the app claim to be used to prevent pregnancy or to conceive? |
ORC_CC02 |
Does the app use body basal temperature (bbt) recorded through an externally connected thermometer? |
ORC_CC03 |
Does the app use rhythm, body basal temperature (bbt) and cervical mucus methods to prevent pregnancy or to conceive? |
ORC_CC04 |
Does the developer claim that the app can be used as a natural method of birth control? |
ORC_CC05 |
Is the app intended to be used for the control of conception? |
ORC_MD06 |
Is the app used in combination with drugs or medication? (e.g. medication reminders) |
ORC_AE20 |
Is the app a companion of the device, as opposed to having been designed to connect with a third party manufacturer's device? |
ORC_F26 |
Monitoring
Question |
Question Reference Source |
Does the app allow the monitoring of key health information? |
ORC_MN01 |
---|---|
Does the app involve the recording of relevant data over time for the user to access and review (with no ‘intelligent’ manipulation of that data by the app)? |
ORC_MN02 |
Does the app involve the automated assessment or interpretation of relevant data to deliver alerts, insights, reminders or adjustments regarding the user’s health or lifestyle? |
ORC_MN03 |
Is the app: |
ORC_MN04 |
Is the output of the app’s monitoring intended to affect the treatment of an individual? |
ORC_MN05 |
Does the app allow others (i.e. not the user) to monitor or view the health data captured? |
ORC_MN06 |
Does the app automatically measure and/or record data about a user’s specified condition, and transmit the data to a professional, caregiver or third party organisation, without any input from the user? |
ORC_MN07 |
Does the app generate any alarms or alerts from the data recorded by the app or a connected device? |
ORC_MN08 |
Are the alarms generated by user-defined filtering rules? |
ORC_MN09 |
What type of intervention or treatment does the app provide? |
ORC_TS11 |
Online Consultations
Question |
Question Reference Source |
Can the app be used for patients to have online consultations, conversations, or related Health Care services with a healthcare professional? |
ORC_F14 |
---|---|
Is this through video consultation? |
ORC_OC02 |
Does the app allow healthcare professionals to provide clinical advice, as opposed to the app providing advice itself? |
ORC_EF09 |
If the app allows healthcare professionals to provide clinical advice through the app, rather than the app providing the advice itself, how does it do this? |
ORC_OC01 |
Administrative Services
Pharmacy
Question |
Question Reference Source |
Does the app allow users to order and request prescriptions? |
ORC_F13 |
---|---|
Reminders/Notifications
Question |
Question Reference Source |
Does the app send push notifications? |
ORC_D29 |
---|---|
Does the app send email notifications? |
ORC_D30 |
External Device
Question |
Question Reference Source |
Is the app's main functionality dependent on the user having one of the devices to connect with the app? |
ORC_F27 |
---|---|
Do any of the features or functions of the app appear to allow it to be used to control a medical device? |
ORC_F30 |
Forums and Contacts
Question |
Question Reference Source |
Are there opportunities to link with other users (buddying, forums or group education)? |
ORC_U19 |
---|---|
Does the app provide an internally hosted forum or online community for their users? |
ORC_FC01 |
Does the app link to a third-party service to host a forum or online community for their users? |
ORC_FC02 |
Does the app allow two-way communication between citisens, patients or healthcare professionals? |
ORC_EF10 |
Goal Setting
Question |
Question Reference Source |
Does the app provide gamification or goal setting features for the user? |
ORC_F06 |
---|---|
Does the app set goals for the user? |
ORC_GS01 |
Does the app allow the user to set goals for themselves? |
ORC_U21 |
Customisation
Question |
Question Reference Source |
Can the app presentation be customised by the user? |
ORC_CUS01 |
---|---|
Does the app respond to preferences in the device? |
ORC_CUS02 |
Business Model
Question |
Question Reference Source |
Is the app totally free? |
ORC_U29 |
---|---|
How is the app funded? |
ORC_BM01 |
Does the app contain advertisements? |
ORC_U27 |
Benefits
Question |
Question Reference Source |
What are the claimed or implied benefits of the App? |
ORC_BF01 |
---|---|
DATA & PRIVACY
Privacy Policy
Initially, the evaluation identifies the relevant privacy policy for the app, which is available to users through the app and/or the App Store or Play Store and/or on the website. The more transparent the privacy policy, the better. Ultimately, the privacy policy must clearly state that user data will not be used or shared with other parties, except as described in the privacy policy, or without express consent of the user. Ideally it will identify:
· what data is collected from the user and how,
· if the user is informed of the developer’s intentions with processing and sharing their data, and
· if the user’s consent is obtained.
The privacy policy should accurately reflect the data usage of the app. The Assessors will be able to note if any data is collected outside of what is detailed in the privacy policy. Additionally, the policy should inform users of the developer’s intent to use their data for marketing purposes. If user data is shared for any other purposes other than basic use of the app, or legal obligations, then the evaluation considers if the user is able to opt out of these activities.
Privacy Policy
Question |
Question Reference Source |
Is there a privacy policy clearly available via the Web App/ Website? |
ORC_D39a |
Is there a privacy summary published anywhere by the developer? |
ORC_D39b |
Is the privacy policy made immediately available when the user first opens the app? |
ORC_DP03 |
Is the privacy policy made available when the user is signing up to the service? |
ORC_DP04 |
Is it published within the app? |
ORC_DP01 |
Is it available externally via the app, or via a linked website? |
ORC_DP02 |
Is it available via the relevant app store? |
ORC_DP05 |
What data does the privacy policy state the developer collects? |
ORC_DP06 |
Is the policy accurate, with regards to the data the developer intends to collect? |
ORC_DP07 |
Does the app state that data collected by the app is stored locally, unless the user manually exports the data? |
ORC_D10a |
How does the developer obtain consent for the processing of user data? |
ORC_DP08 |
Does the privacy policy provide the name and contact details of their Data Protection Officer (DPO), or similar individual representative for the company? |
ORC_DP14 |
Provide the details of the DPO: (Text Response) |
ORC_DP15 |
Data use
Once it is established what data is collected by the app, the evaluation looks at how that data is used and shared, and if this is communicated to the user. The privacy policy should state all intended uses and legal basis of processing user data, such as legal obligation, research or marketing. Users should also be given the option to withdraw consent for the use of their data, particularly for marketing.
Question |
Question Reference Source |
Does the developer fully inform the user of how they will collect data about them? |
ORC_D69 |
Does the developer provide users with details on all the purposes of processing user data? |
ORC_D13 |
What is automatically shared data used for? |
ORC_DP10 |
Does the developer appear to intend to share or process the user data collected by the app for any purposes that have not been made clear to the user, or for any purposes they deem necessary? |
ORC_D38 |
Does the developer inform users that they would like to use their data for the purpose of marketing? |
ORC_D71 |
Does the developer obtain informed consent separately, for the purpose of marketing? |
ORC_DP12 |
Is the user informed of how they can opt out of each of these activities? |
ORC_D28 |
If the user can not opt out of all processing activity, does the developer clearly explain which activities they cannot opt out of and why? |
ORC_DP13 |
Is the user informed that their data will not be shared with other parties, except for the purposes that have been set out in the privacy policy? |
ORC_D16 |
Data Storage and Transit/Transfer
The key areas in this section are surrounding data storage and data transfer. The data privacy policy should inform the user of where their data is stored, how their data is protected in storage, and how it is protected in transit between the user’s device and the host storage. The NordDEC looks for specific and secure storage techniques, such as encryption or firewalls. During transit, it is preferable that data is protected using SSL encryption.
Question |
Question Reference Source |
Does the data privacy policy or equivalent provide detail about where the data collected by the app will be stored (i.e. on the app or in an external data warehouse, cloud server etc.)? |
ORC_DST01 |
Where is the data stored? |
ORC_DST02 |
Does the data privacy policy, or equivalent, state whether personal data is stored using recognised secure data storage technologies? |
ORC_DST03 |
Is all personally identifiable data encrypted in transit between the device and any external host storage? |
ORC_D17 |
Is the user informed that online video consultations use secure encryption methods? |
ORC_DST04 |
Data Standards and Management
The NordDEC will award additional points if an app developer is compliant with any recognised International Data Management Standards such as ISO 27001. The privacy policy should inform users of a data retention period, and a method for data destruction. The NordDEC also identifies whether the developer has a policy in place to deal with any data security breaches.
Question |
Question Reference Source |
Does the policy state its compliance with recognised International Data Management Standards? |
ORC_DM01 |
Does the policy contain details of the length of time data is retained? |
ORC_D19 |
Is there a statement containing details of a method for data destruction? |
ORC_D20 |
Is there a statement that sets out a process for managing data confidentiality breaches? |
ORC_D21 |
General Data Protection Regulation (GDPR)
This evaluation area focuses on the General Data Protection Regulation, which in May 2018 came into force. The NordDEC is concerned that all apps, particularly those developed in the EEA, are fully compliant with the GDPR. This means a clear and explicit statement of compliance, as well as confirming that the user is entitled to the 7 user rights.
The developer should also inform the user of how they can exercise these rights, and should commit to responding within a time frame of 2 months or less. Under the GDPR, the policy should outline the legal basis for collection of user data, and ensure that only minimal data is collected from the user.
All question relating to GDPR will only be asked for apps that collect and process personal and/or sensitive data.
Question |
Question Reference Source |
Is there a statement that confirms the App’s compliance to GDPR 2018? |
ORC_D23 |
Is the user informed of the legal basis for which data is collected from them? |
ORC_D60 |
Is the user informed of the developer’s intent to ensure that data minimisation principles are met? |
ORC_DPR03 |
Is there a statement that the policy will be updated duly should the purpose of data collection change? This may mean re-obtaining consent (if consent was the lawful basis). |
ORC_D61 |
Are users informed of their rights with regards to their data? Are users clearly informed of the individual privacy rights they are entitled to expect under GDPR? |
ORC_DPR01 |
Has the developer made the existence of the data subject’s right to request that their personal data is deleted clear? |
ORC_D93 |
Has the developer made the existence of the data subject’s right to access their personal data clear? |
ORC_D25 |
Has the developer made the existence of the data subject’s rights to rectify their personal data clear? |
ORC_D56 |
Has the developer made the existence of the data subject’s rights to restrict the use of their personal data clear? |
ORC_D81 |
Has the developer made the existence of the data subject’s rights to object to the processing of their personal data clear? |
ORC_D57 |
Has the developer made the existence of the data subject’s rights to portability of (receive) their personal data clear? |
ORC_D59 |
Has the developer made the existence of the data subject’s right to withdraw consent for the use of their personal data clear? |
ORC_D58 |
Has the developer made clear the existence of the user’s right to request that they are not subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her? |
ORC_DPR02 |
Does the developer provide details which the user can contact them on to exercise their rights? |
ORC_D82 |
Is the user informed of the time frame in which the developer will respond to any requests to exercise their rights? |
ORC_D83 |
Other Data Questions
This section also looks into children’s data use (if applicable), or if a user can report knowledge of a child accessing the app without parental consent. The transparency of the privacy policy should extend to inform the user that any links to third party websites or apps are not covered by the developer’s privacy policy, and users should make themselves aware of such third party policies. The privacy policy should contain contact details, should the user wish to make further enquiries regarding their data. The NordDEC also awards additional value points if the app provides the user with an additional, optional layer of security to protect their data.
Question |
Question Reference Source |
Are users clearly informed of the use of cookies when first landing on the developers site/app? |
ORC_D99 |
Are user's required to confirm their acceptance of the developer's use of cookies, when initially informed of the use? |
ORC_D100 |
Does the developer provide a full Cookie Policy, separate from the Terms of Service and/or Privacy Policy? |
ORC_D84 |
Is the app ‘particularly likely’ to be used by children, even if they are not the primary market for the product? |
ORC_D44 |
Are users informed of how they can report, to the developer, any knowledge of a child accessing the app and providing personal data, without parental consent? |
ORC_DO01 |
Is the user made aware that by following links to third party websites, the developer’s policies no longer apply, and that the user should make themselves aware of the third party’s policies? |
ORC_D91 |
Is the user informed of how they can make further enquiries about the company’s privacy policy? |
ORC_D92 |
Does the app allow the user to set their preferences for sharing the app data with or from other apps (e.g. Facebook / Instagram/Fitbit etc)? |
ORC_D06 |
Is there functionality within the app to allow the user to set their preferences for sharing app data with others users (clinicians, carer, family, friends, buddies)? |
ORC_D27 |
Is it strictly necessary for anyone to easily access the personal information that persists on the app? e.g. to access health info during an emergency. |
ORC_DO02 |
Are users provided options to introduce additional security measures to protect their data on the app? eg. set additional pass codes for access to the app, after accessing the device is unlocked. |
ORC_DO03 |
Does the app use a sign up/sign in verification/authentication model |
ORC_DO04 |
What type of model is being used? (please describe) |
ORC_DO05 |
Enhanced Data Evaluation
Question |
Question Reference Source |
Is the developer of the app the Data Controller, Data Processor or Product Manufacturer Only? |
ORC_ERC_EDC_O01 |
Is the developer subject to EU Data Protection Laws? |
ORC_ERC_EDC_O02 |
Provide EU data protection registration details: |
ORC_ERC_EDC_O03 |
If this information is unavailable, please explain why? |
ORC_ERC_EDC_O04 |
Is the developer of the App a public authority or body? |
ORC_ERC_EDC_DPO01 |
Do any of the developer’s processing activities include automated systematic and extensive profiling with significant effects? |
ORC_ERC_EDC_DPA01 |
Do any of the developer’s processing activities include large scale use of sensitive/special category data? |
ORC_ERC_EDC_DPA02 |
Do any of the developer’s processing activities include systematic monitoring of a publicly accessible area on a large scale? |
ORC_ERC_EDC_DPA03 |
Do any of the developer’s processing activities include any of the following indicators of high risk processing? |
ORC_ERC_EDC_DPA04 |
Is the developer required to have a Data Protection Officer? |
ORC_ERC_EDC_DPO06 |
Is the developer required to have written a Data Protection Impact Assessment (DPIA)? |
ORC_ERC_EDC_DPIA01 |
Has the developer submitted a DPIA or made a DPIA publicly available? |
ORC_ERC_EDC_DPIA02 |
Has the developer, at a minimum, published details on how to communicate with the DPO, for individuals to contact the DPO as needed? |
ORC_ERC_EDC_DPO02 |
Is there evidence that the DPO has the necessary professional qualities, and in particular, experience and expert knowledge of data protection law? |
ORC_ERC_EDC_DPO04 |
Does the DPO hold a position within the organisation that may lead him or her to determine the purposes and the means of the processing of personal data, or require him or her to engage in further tasks and duties that may result in a conflict of interests with the primary tasks of a DPO? |
ORC_ERC_EDC_DPO05 |
If a DPIA has been submitted, does the document provide details of a Data Protection Officer? |
ORC_ERC_EDC_DPO03 |
Does the DPIA describe how the developer will collect data about individuals? |
ORC_ERC_EDC_DPA05 |
Does the DPIA describe how the developer will store the data collected? |
ORC_ERC_EDC_DPA06 |
Does the DPIA describe what the data collected by the developer will be used for? |
ORC_ERC_EDC_DPA07 |
Does the DPIA provide details of who will have access to the data collected? |
ORC_ERC_EDC_DPA08 |
Does the DPIA provide details of who the developer will share data with (any third parties)? |
ORC_ERC_EDC_DPA09 |
Does the information regarding data sharing include details of the developer’s use of any processors? |
ORC_ERC_EDC_DPA10 |
Does the DPIA provide details of their own data retention periods? |
ORC_ERC_EDC_DPA11 |
Does the DPIA provide details of the length of time each third party recipient of data will retain data for? |
ORC_ERC_EDC_DPA12 |
Does the DPIA provide details of the security measures that have been put in place in order to protect the data being processed? |
ORC_ERC_EDC_DPA13 |
Does the DPIA provide details of the developer using any new technologies? |
ORC_ERC_EDC_DPA14 |
Does the DPIA describe any novel types of processing that the developer is using? |
ORC_ERC_EDC_DPA15 |
Does the DPIA describe which screening criteria they have identified/flagged as high risk? |
ORC_ERC_EDC_DPA16 |
Does the DPIA describe the nature of the personal data being processed? |
ORC_ERC_EDC_DPA17 |
Does the DPIA describe the volumes and variety of personal data that is being processed? |
ORC_ERC_EDC_DPA18 |
Does the DPIA describe the sensitive nature of any items of personal data being processed? |
ORC_ERC_EDC_DPA19 |
Does the DPIA provide details of the extent and frequency of the processing? |
ORC_ERC_EDC_DPA20 |
Does the DPIA provide details on the duration of any processing activities covered by the DPIA? |
ORC_ERC_EDC_DPA21 |
Does the DPIA detail the number of data subjects involved in the processing activities described? |
ORC_ERC_EDC_DPA22 |
Does the DPIA provide details of the geographical area covered by the processing activities described? |
ORC_ERC_EDC_DPA23 |
Does the DPIA identify all sources of the data being collected? |
ORC_ERC_EDC_DPA24 |
Does the DPIA describe the nature of the developer’s relationship with the individuals whose data is being processed? |
ORC_ERC_EDC_DPA25 |
Does the DPIA describe how far individuals have control over their data? |
ORC_ERC_EDC_DPA26 |
Does the DPIA describe how far individuals are likely to expect the processing of their data to occur? |
ORC_ERC_EDC_DPA27 |
Does the DPIA provide details on whether the data subjects concerned include children or other vulnerable individuals? |
ORC_ERC_EDC_DPA28 |
Does the DPIA identify any previous experience that the developer has in dealing with the intended type of processing? |
ORC_ERC_EDC_DPA29 |
Does the DPIA identify any relevant advances in the technology and/or security being used in the processing? |
ORC_ERC_EDC_DPA30 |
Does the DPIA identify any current issues of public concern with regards to the intended processing? |
ORC_ERC_EDC_DPA31 |
Does the DPIA detail compliances with any UK GDPR codes of conduct or UK certification schemes? |
ORC_ERC_EDC_DPA32 |
Does the DPIA detail whether the developer has considered and complied with relevant codes of practice? |
ORC_ERC_EDC_DPA33 |
Does the DPIA provide details of the developer’s legitimate interests, with regards to the purpose of processing data? |
ORC_ERC_EDC_DPA34 |
Does the DPIA detail the legal basis/bases upon which the developer relies for processing the data collected? |
ORC_ERC_EDC_DPA35 |
Does the DPIA provide details of the intended outcomes of the data processing for the individuals concerned? |
ORC_ERC_EDC_DPA36 |
Does the DPIA provide details of the anticipated benefits of the processing for the developer or for society as a whole? |
ORC_ERC_EDC_DPA37 |
Has the developer sought and documented the views of the individuals whose data will be processed, or their representatives? |
ORC_ERC_EDC_DPA38 |
Does the DPIA provide a justifiable reason for not carrying out consultation with individuals or their representatives? |
ORC_ERC_EDC_DPA39 |
Does the DPIA provide details of consultation with any third party data processors who will be involved in the processing of user data? |
ORC_ERC_EDC_DPA40 |
Does the DPIA provide details of consultation with all relevant internal stakeholders? |
ORC_ERC_EDC_DPA41 |
Does the DPIA provide details of any advice being sought from other independent experts? |
ORC_ERC_EDC_DPA42 |
Does the DPIA include details on how the developers plans will help achieve their purpose? |
ORC_ERC_EDC_DPA43 |
Does the DPIA provide evidence that the developer has considered whether there are any other reasonable ways to achieve the same result? |
ORC_ERC_EDC_DPA44 |
Does the DPIA include details on how the developer will prevent function creep? |
ORC_ERC_EDC_DPA45 |
Does the DPIA include details on how the developer will ensure data quality? |
ORC_ERC_EDC_DPA46 |
Does the DPIA include details on how the developer will ensure data minimisation? |
ORC_ERC_EDC_DPA47 |
Does the DPIA include details on how the developer intends to provide privacy information to individuals? |
ORC_ERC_EDC_DPA48 |
Does the DPIA include details on how the developer intends to implement and support individuals’ rights? |
ORC_ERC_EDC_DPA49 |
Does the DPIA include details on the measures that will be taken to ensure any processors comply with their obligations? |
ORC_ERC_EDC_DPA50 |
Does the DPIA include details of the safeguarding measures that have been put in place for international transfers of data? |
ORC_ERC_EDC_DPA51 |
Has the developer detailed, in the DPIA, all the risks that they have identified relating to each of their processing activities and those relating to third party processors? |
ORC_ERC_EDC_DPA52 |
Are there any additional risks, that you have been able to identify, which have not been included in the DPIA? (please detail) |
ORC_ERC_EDC_DPA53 |
Does the DPIA include an assessment of potential security risks? |
ORC_ERC_EDC_DPA54 |
Does the assessment of security risks include the sources of the risks and the potential impacts of the each type of breach? |
ORC_ERC_EDC_DPA55 |
Has an objective approach to assessing the risks been taken through the use of a structured risk matrix that takes into account both the likelihood of harm and the severity of the impact? |
ORC_ERC_EDC_DPA56 |
Has the developer used a different, but acceptable structured approach to objectively assessing the risks associated with their processing activities? |
ORC_ERC_EDC_DPA57 |
Has the developer also considered their own corporate risk, such as the impacts of regulatory action, reputational damage of loss of public trust? |
ORC_ERC_EDC_DPA58 |
Has the developer detailed the source of each risk that they have identified? |
ORC_ERC_EDC_DPA59 |
Has the developer identified the measures that they will put in place in order to mitigate each of the identified risks? |
ORC_ERC_EDC_DPA60 |
Where mitigations have not been put in place, has the developer provided justified reasons for not doing so? |
ORC_ERC_EDC_DPA61 |
Are there any additional mitigations, that you have been able to identify as achievable, that the developer does not appear to have considered? (please detail). |
ORC_ERC_EDC_DPA62 |
Does the DPIA contain details of the additional measures that the developer planned on taking? |
ORC_ERC_EDC_DPA63 |
Does the DPIA contain a record on whether each of the risks have been removed, reduced, or accepted? |
ORC_ERC_EDC_DPA64 |
Does the DPIA provide details of the overall residual risk, after taking additional measures? |
ORC_ERC_EDC_DPA65 |
Have any of the residual risks been assessed as being “high risk”? |
ORC_ERC_EDC_DPA66 |
Does the DPIA provide details that the need to consult the ICO has been considered? |
ORC_ERC_EDC_DPA67 |
Does the DPIA provide details or evidence of the advice of the DPO being sought, as part of the sign-off process? |
ORC_ERC_EDC_DPA68 |
If the developer has decided not to follow the advice provided by the DPO, have they recorded their reasons? |
ORC_ERC_EDC_DPA69 |
Does the app process personal/sensitive Social Care data? |
ORC_ERC_EDC_NHS01 |
Does the developer collect usage or bug report data? |
ORC_ERC_EDC_UBRO01 |
Is this collected through informed consent? |
ORC_ERC_EDC_UBRO02 |
Is this data fully anonymised? |
ORC_ERC_EDC_UBRO03 |
Does the organisation collect data, through the app, using cookies, web beacons or other similar technologies? |
ORC_ERC_EDC_CK01 |
Is there a cookie policy? |
ORC_ERC_EDC_CK02 |
Is there a cookie policy provided separate from the terms and condition and privacy policy? |
ORC_ERC_EDC_CK03 |
Are users made aware of the use of strictly necessary cookies? |
ORC_ERC_EDC_CK04 |
Is user consent obtained for the use of non-strictly necessary cookies? |
ORC_ERC_EDC_CK05 |
Does the organisation keep a log of user consent? (eg. evidence of when consent was obtained and the information provided at the time of consent) |
ORC_ERC_EDC_CK06 |
Are users informed of how they can easily opt out of the use of cookies? |
ORC_ERC_EDC_CK07 |
Is the product aimed at children or likely to be used by children? |
ORC_ERC_EDC_COP01 |
Where consent was the legal basis for processing data was consent, at the time the an individual was a child, then requests for the erasure of data are complied with, whenever possible? |
ORC_ERC_EDC_COP04 |
Have children been consulted when designing this processing practice? |
ORC_ERC_EDC_COP05 |
Has the privacy policy been written in plain, age appropriate language? |
ORC_ERC_EDC_COP06 |
Is consent sought from a responsible parent/guardian? |
ORC_ERC_EDC_COP07 |
Does the developer ensure they do not seek parental/guardian consent when providing online preventive or counselling services to children? |
ORC_ERC_EDC_COP08 |
Are there two separate versions of privacy policies, one aimed at the child and the other at the responsible parent/guardian? |
ORC_ERC_EDC_COP09 |
When marketing the product outside of their country of residence, has the developer taken into consideration other jurisdictional laws regarding children’s privacy (eg. age restrictions)? |
ORC_ERC_EDC_COP10 |
Has the DPIA been completed with specific details on the assessed risks to children and the mitigations in place? |
ORC_ERC_EDC_COP03 |
Has a process been designed and put in place that allows children to easily access, understand and exercise their own data protection rights? |
ORC_ERC_EDC_COP02 |
Is data shared with and processed by any third parties? |
ORC_ERC_EDC_PC01 |
What services do the third parties provide? |
ORC_ERC_EDC_PC02 |
Is any of the shared data personally identifiable or sensitive? |
ORC_ERC_EDC_PC03 |
Are there written binding contract agreements between the organisation (controller) and each third party (processor)? |
ORC_ERC_EDC_PC04 |
Has the developer provided a copy/draft/template of these written agreements? |
ORC_ERC_EDC_PC05 |
Does the contract clearly define the subject matter and duration of the processing? |
ORC_ERC_EDC_PC06 |
Does the contract clearly define the purpose of the processing? |
ORC_ERC_EDC_PC07 |
Does the contract clearly set out the categories of data subject and the types of data that will be processed? |
ORC_ERC_EDC_PC14 |
Does the contract clearly describe the obligations and rights of the controller? |
ORC_ERC_EDC_PC08 |
Does the contract make clear that the processor must only act upon written instruction from the controller? |
ORC_ERC_EDC_PC09 |
Does the contract make clear to the processor their responsibilities in ensuring that their employees are subject to a duty of confidence? |
ORC_ERC_EDC_PC10 |
Does the contract make clear to the processor their responsibilities in ensuring all appropriate safeguarding measures are in place to ensure the security of the data they are processing? |
ORC_ERC_EDC_PC11 |
Does the contract clearly state that the processor must assist the controller when responding to requests to exercise user rights? |
ORC_ERC_EDC_PC12 |
Does the contract clearly state that the processor must assist the controller in meeting their legal data protection requirements? (e.g. notifying of breaches and completing DPIAs). |
ORC_ERC_EDC_PC13 |
CLINICAL EVIDENCE
The Evidence Standards for Digital Health Technologies Framework (“ESF”) was created by the UK’s National Institute for Health and Care Excellence (“NICE”). This framework clustered app’s into relevant Tiers and identified for each Tier what forms of ‘evidence’ or ‘assurance’ would be required. It is therefore better to think of the ESF as an Assurance Standards Framework, with evidence being just one of many elements within that digital assurance matrix.
An adapted version of the ESF has been developed over time with and has now been adopted in numerous other national and pan-national Digital Health Assessment Frameworks in areas like New Zealand, Canada, Israel and the Netherlands. We conduct an analysis of any evidence available through the Review Resources. If this exists, the app is evaluated against a series of questions to determine the quality of this evidence. We look for:
· a suitable sample size and make up;
· a p value of below 0.05 to indicate significance;
· a p value below 0.02 for near significance; and
· an appropriate comparator.
This is scaled against the NICE Evidence Standards Framework and we look for a higher level of evidence for apps with more complex functionality and higher risk.
Question |
Question Reference Source |
EE02: What type/s of evidence is available? Survey, RCT, Pilot study, Observational (Case study, Cross-sectional, Cohort), Meta-Analysis/Systematic Review |
ORC_EE02 |
How many pieces of evidence does the app provide? |
ORC_EE14 |
How many RCT's and/or observational studies does the app have? |
ORC_EE13 |
For each type of relevant evidence: EE10: What category does the evidence relate to? |
ORC_EE10 |
For each type of relevant evidence: EE11: What benefit does the evidence relate to? |
ORC_EE11 |
EE03: Provide links to the publicly available evidence/published evidence that the developer has provided. |
ORC_EE03 |
For each type of relevant evidence: EE04: Is the sample size appropriate? |
ORC_EE04 |
For each type of relevant evidence: EE05: Does the evidence found provide a p-value? |
ORC_EE05 |
For each type of relevant evidence: EE06: Does the p-value demonstrate significance (p<0.05)? |
ORC_EE06 |
For each type of relevant evidence: EE12: Does the p-value demonstrate near significance (p<0.2)? |
ORC_EE12 |
For each type of relevant evidence: EE07: Is there a comparator? |
ORC_EE07 |
For each type of relevant evidence: EE08: Is the comparator validated? |
ORC_EE08 |
Behavioural Change
There are some scenarios where the app utilises widely accepted techniques with a breadth of evidence. In this instance the app may not deem it appropriate to fund a full randomised control trial to demonstrate effectiveness. Therefore we give some value points for fully referencing evidence for behavioural change techniques used within the app. This is not however treated in the same way as where the app has provided direct evidence of its own effectiveness.
Question |
Question Reference Source |
Does the App have its own high quality study? |
ORC_BCT01 |
Does the App reference and evidence its behaviour change technique? |
ORC_BCT02 |
Professional Backing
We look for evidence of an appropriate professional being involved in the app's design and development, or if the app has been externally accredited. A relevant professional is deduced in the context of an app. For example, for a simple yoga app we would accept a qualified yoga instructor as a relevant professional, but for a complex clinical solution we would only accept a relevantly qualified clinician. External accreditations are wide ranging, but we would look for an appropriate body, for example the British Heart Foundation giving an endorsement to a cardiology app.
Question |
Question Reference Source |
Is there a suitably qualified Professional involved in the development team of the App? |
ORC_PB01 |
Does the organisation behind the App have relevant credentials? |
ORC_PB02 |
Is there evidence of an endorsement by a relevant body? |
ORC_PB03 |
Are organisations using the App? |
ORC_PB04 |
Is there a statement that it has been positively evaluated or validated by a relevant healthcare professional? |
ORC_PB05 |
Please specify who the relevant experts are and what qualifications they hold. |
ORC_AE17· |
Is there evidence within the app that the developer has validated any Guidance/Context with relevant reliable information sources or references? |
ORC_PB06 |
ORCHA Adapted ESF Compliance
The first part of this section assesses which ESF Tier the app falls under, and is non-scoring. The second part assesses whether the app meets the minimum requirements of that Tier. Compliance with the ESF is determined by the app answering positively to all questions that have been flagged as a requirement for its Tier of the ESF and all Tiers below.
Question |
Question Reference Source |
What Tier of the ESF is the App? |
|
Is the app Tier 1? |
ORC_ESF01 |
Is the app Tier 2a? |
ORC_ESF02 |
Is the app Tier 2b? |
ORC_ESF03 |
Is the app Tier 3a? |
ORC_ESF04 |
Is the app Tier 3b? |
ORC_ESF05 |
Has the App met: |
|
Tier 1 minimum requirements? |
ORC_ESF06 |
Tier 2a requirements? |
ORC_ESF07 |
Tier 2b requirements? |
ORC_ESF08 |
Tier 3a requirements? |
ORC_ESF09 |
Tier 3b requirements? |
ORC_ESF10 |
Does the app have appropriate evidence for the ESF tier? |
ORC_ESF11 |
Medical Devices
It is proposed that the NORDIC Assessment assess if the app is likely to be a medical device under the current Guidance/Context from the MDR (https://ec.europa.eu/growth/sectors/medical-devices_en ). We then evaluate if the app displays the relevant CE mark.
Question |
Question Reference Source |
Is the app a medical device? |
ORC_MD11 |
Does the app have a CE mark? |
ORC_AE06 |
Does the app state that it has been assessed by the MHRA or other relevant body, and does not require a CE mark? |
ORC_AE08 |
What class is the app certified as? |
ORC_MD09 |
Has the app been FDA approved? (Food and Drug Administration) |
ORC_FDA01 |
Has the app been FDA cleared? |
ORC_FDA02 |
Clinical Safety/Risk Management
It is proposed that the evaluation looks for any safeguarding measures in communication functions of the app, if relevant.
Question |
Question Reference Source |
Is there a statement or any evidence showing that appropriate safeguarding measures are in place around peer-support and other communication functions within the platform?
|
ORC_AE13 |
Does the Developer clearly identify who the app should and should not be used by? |
ORC_S01 |
Does the Developer publish their risk management processes? |
ORC_S02 |
Does the Developer make clear risks associated with using the app? |
ORC_S03 |
Is there a way for the user to confirm that the data input is accurate? |
ORC_S04 |
Does the developer list a Clinical Safety Officer (CSO) on/in any relevant sites/content? |
ORC_S05 |
Please provide more detail. |
ORC_S06 |
Is the App/Solution in scope for a clinical safety assessment? |
ORC_ERC_OCSA_CSS1 |
Why is the app in scope? |
ORC_ERC_OCSA_CSS2 |
Has the Developer provided a thorough summary about why the App is out of scope? (Eg is it complete? Does it match the functionality put forward?) |
ORC_ERC_OCSA_CSS3 |
What risks if any, have been documented concerning harm to a patient? |
ORC_ERC_OCSA_CSS4 |
If the App/Solution has been deemed in scope, has the developer supplied suitable Risk Management Documentation? |
ORC_ERC_OCSA_CSS5 |
Please confirm the name of your Clinical Safety Officer (CSO), their profession and registration details? |
ORC_DTAC_CS120 |
Is the CSO a suitably qualified and experienced clinician? |
ORC_ERC_OCSA_CSO1 |
Does the named CSO have appropriate qualifications and up to date registration details? |
ORC_DTAC_CS121 |
Is there evidence the CSO has played an active part in the clinical safety process – approval of the risk management file, hazard assessment participation etc? |
ORC_ERC_OCSA_CSO3 |
Has the developer provided their Clinical Safety Case and Hazard Log? |
ORC_DTAC_CS112 |
Does the Developer outline the need for the Risk Management Documentation? |
ORC_ERC_OCSA_CSD5 |
Does it have full version history and issue date published? |
ORC_ERC_OCSA_CSD2 |
Has the Developer described their clinical risk management system? ( identification of key personnel, their roles and responsibilities; identification of clinical risk management governance structure. |
ORC_ERC_OCSA_CSD6 |
Does the Safety Case make a mention of a test summary? (Summary of any outstanding test issues and the impact on clinical safety) |
ORC_ERC_OCSA_CSD7 |
Does the Safety Case have a summary statement showing sign off from the CSO? |
ORC_ERC_OCSA_CSD8 |
Is there evidence of a CSO reviewing, contributing or approving the Safety Case? |
ORC_ERC_OCSA_CSD3 |
Does the Hazard Log have full version history and issue date published? |
ORC_ERC_OCSA_CSD9 |
Are the hazards listed complete? (From a review of the listed hazards, do they have all of the details required completed such as name, clinical impact and risk ratings. Do the risk ratings look appropriate or do they appear to be copy and paste throughout the listed hazards? Are they scored on the low side? Does the consequence change pre and post assessment? |
ORC_ERC_OCSA_CSD4 |
Are the potential harms related to the user/patient? |
ORC_ERC_OCSA_CSD11 |
Do the harms outline what the clinical impact may be for the user? |
ORC_ERC_OCSA_CSD12 |
Has the Developer covered all of the possible causes for each Hazard? |
ORC_ERC_OCSA_CSD13 |
Do the risk ratings look appropriate or do they appear to be copy and paste throughout the listed hazards? |
ORC_ERC_OCSA_CSD14 |
Are hazards split incorrectly into potential and actual harm? |
ORC_ERC_OCSA_CSD15 |
Is there evidence of a CSO reviewing, contributing or approving the hazard log? |
ORC_ERC_OCSA_CSD10 |
Has the Developer implemented the clinical risk analysis activities defined in the Clinical Risk Management Plan? |
ORC_ERC_OCSA_CRAP1 |
Is the Clinical Risk Analysis carried out by a multi-disciplinary group? |
ORC_ERC_OCSA_CRAP2 |
Has the Developer defined the clinical scope of the Health IT System which is to be delivered? |
ORC_ERC_OCSA_CRAP3 |
Has the Developer defined the intended use of the Health IT System which is to be delivered? |
ORC_ERC_OCSA_CRAP4 |
Does the Risk Management Documentation make it clear where the App fits into the Clinical Workflow – How would a patient use the app appropriately to become well again? |
ORC_ERC_OCSA_CRAP5 |
Has the Developer outlined Third Party Products integrated within the Health IT System to be released? |
ORC_ERC_OCSA_DSP1 |
Has the Developer deploying the Health IT System considered how it will impact on the current business processes and ways of working? |
ORC_ERC_OCSA_DSP2 |
Is there usability and human factors related evidence within the scope? |
ORC_ERC_OCSA_DSP3 |
Has the Developer assessed any infrastructure at the Health Organisation that is within their scope of influence, required to support the deployment of the Health IT System? (This may be achieved by the Manufacturer specifying the minimum system requirements) |
ORC_ERC_OCSA_DSP4 |
Where data migration is to be undertaken by the Developer it should be included in the scope of the clinical risk management activities. |
ORC_ERC_OCSA_DSP5 |
Is the Data Migration being undertaken by the Developer properly covered in the documentation? |
ORC_ERC_OCSA_DSP6 |
Has the Developer identified any hazards associated with the data migration, analysed and suitably mitigated? (working in conjunction with the relevant Health Organisation as appropriate) |
ORC_ERC_OCSA_IPH1 |
Has the Developer considered the end to end clinical process, including functionality and how that functionality is used? |
ORC_ERC_OCSA_IPH2 |
Has the Developer considered inter and intra Health IT System messaging? |
ORC_ERC_OCSA_IPH3 |
Has the Developer assessed the health IT system architecture and design? |
ORC_ERC_OCSA_IPH4 |
Is there a clear matrix, which is used to define the risk ratings? |
ORC_ERC_OCSA_CRE11 |
For each identified hazard, has the Developer evaluated whether the initial clinical risk is acceptable? |
ORC_ERC_OCSA_CRE9 |
Has the Developer used the risk acceptability criteria previously defined? |
ORC_ERC_OCSA_CRE10 |
Has the Developer identified appropriate clinical risk control measures to remove any unacceptable clinical risk? |
ORC_ERC_OCSA_CRE1 |
Has the Developer assessed Proposed clinical risk control measures to determine whether new hazards will be introduced as a result of the measures? |
ORC_ERC_OCSA_CRE2 |
Has the Developer assessed proposed clinical risk control measures to determine whether the clinical risks for previously identified hazards will be affected? |
ORC_ERC_OCSA_CRE3 |
Is the Developer managing new hazards, or increased clinical risks? |
ORC_ERC_OCSA_CRE4 |
For each identified hazard, has the Developer evaluated whether the residual clinical risk is acceptable? |
ORC_ERC_OCSA_CRE5 |
Has the Developer used the risk acceptability criteria previously defined? |
ORC_ERC_OCSA_CRE6 |
If the residual clinical risk is unacceptable, has the developer identified additional clinical risk control measures in order to reduce the clinical risk? |
ORC_ERC_OCSA_CRE7 |
If the Developer has determined that no suitable risk control measures are possible, have they conducted a clinical risk benefit analysis of the clinical risk? |
ORC_ERC_OCSA_CRE8 |
Has the Developer’s analysis shown that the clinical benefits of the intended use outweigh the residual clinical risk? |
ORC_ERC_OCSA_CRB1 |
Has the Developer implemented the clinical risk control measures identified? (except where these are to be implemented by another organisation.) |
ORC_ERC_OCSA_CRCM1 |
Have the clinical risks from all identified hazards been considered and accepted? |
ORC_ERC_OCSA_CRCM2 |
Have any hazard rating reductions been fully justified? |
ORC_ERC_OCSA_CRCM3 |
Usability & Accessibility
Design and Development
This considers the design and development of the app and whether it follows any recognised app design standards, such as WC3, WCAG 2.0 AA, WCAG 2.1 AA, ISO 9241, Apple HIG, or Android App Quality Guidelines. The NordDEC also considers whether there was any user involvement during the development of the app, user involvement in testing, or if any features were based on user feedback.
Question |
Question Reference Source |
Is there a statement within the app outlining compliance with any currently recognised app design standards? |
ORC_DE01 |
Is there a statement about user feedback during design/development? |
ORC_DT01 |
Is there any evidence of user involvement in testing? |
ORC_DT02 |
(Web-Apps Only) |
ORC_U01a |
Accessibility
Accessibility is important to consider, as the app should be accessible to all users regardless of their specific needs. The NordDEC considers whether the app is customisable to suit certain needs, such as poor sight or hearing impairments. If the app uses any specialist or medical terms, these should be clearly explained to the user.
Question |
Question Reference Source |
Can the user change the font size in-app/does the app respond to device preferences? |
ORC_U04 |
Does the app provide support for users with poor sight? |
ORC_U07 |
Does the app provide support for users with hearing difficulty? |
ORC_U08 |
(Web-Apps Only) |
ORC_U26 |
Usability
This also ties in to the accessibility of the app, including further customisation options. The NordDEC identifies if the app has any functions to aid navigation, such as a home button, back button, help button or search feature. If the app utilises push or email notifications, the NordDEC identifies whether the user has options to manage these for their own preference or privacy, both at app level and at device level. Finally, if there are any bugs identified during evaluation, this will be flagged. If the app contains a forum, then we look for a statement to ensure that forum content is moderated.
Question |
Question Reference Source |
Can the user change the presentation theme? |
ORC_U06 |
Does the app include the following functions: · Home/Menu button · Back button · Help/About button · Search button |
ORC_U32 |
Are any medical, specialist or technical terms explained clearly to the user? |
ORC_U15 |
Does the app send push notifications? |
ORC_D29 |
Does the app send email notifications? |
ORC_D30 |
Does the user have options to manage the notification settings (push/email) within the app for convenience/privacy? |
ORC_D31 |
Does the app inform the user how to manage notification settings for convenience/privacy (to prevent info being shown if device is locked but on show)? (android only question) |
ORC_D32 |
Was there any evidence of bugs during evaluation? |
ORC_U23 |
Support
Support is a key area of this section, as it is important that users are informed of ways in which they can contact the developer should they have any problems or questions with the app. The NordDEC also identifies what type of support is offered to users, and if there is a commitment from the developer to respond to any user queries. We would expect to see that the type of support offered is appropriate to the app level - a higher level app would therefore require a more sophisticated offer of user support.
Question |
Question Reference Source |
If there is a forum, is there a statement within the app that the forum content is moderated? |
ORC_FC03 |
Is there a statement about how to report issues to the developer? |
ORC_U24 |
What kind of support is offered? |
ORC_U33 |
Is there any statement within the app about the developer’s commitment to addressing problems reported to them? (e.g. timescales to respond, commitment to eradicate reported bugs and faults) |
ORC_U25 |
Security & Technical Stability
Technical Stability
Question |
Question Reference Source |
Does the App connect to an internet-based API (e.g. App Developer Web Service, Social Media, Adverts)? |
ORC_ERC_OTS_C01 |
List the APIs |
ORC_ERC_OTS_C02 |
Does the App connect to a medical device? |
ORC_ERC_OTS_C03 |
Does the App connect to healthcare services? |
ORC_ERC_OTS_C04 |
Does the App operate without wi-fi? |
ORC_ERC_OTS_C05 |
Does the App operate without cellular network? |
ORC_ERC_OTS_C06 |
Is the platform Web based or Mobile? |
ORC_ERC_OTS_D04 |
Does the App access, process or store Personal and/or Sensitive Data? |
ORC_ERC_OTS_D01 |
Is sensitive data persisted to the mobile device? |
ORC_ERC_OTS_D02 |
Does the App access, process or store Personal and/or Sensitive Data? |
ORC_ERC_OTS_D03 |
What Permissions does the App request? |
ORC_ERC_OTS_P01 |
Does the App provide Alerts or Notifications? |
ORC_ERC_OTS_OTF01 |
Does the App provide Suggestions? |
ORC_ERC_OTS_OTF02 |
Does the App undertake calculations? |
ORC_ERC_OTS_OTF03 |
Are the source code and any configuration items for the product version controlled with all changes audited? |
ORC_ERC_OTS_PSL01 |
Provide details of any associated processes / procedures and tools that are used. |
ORC_ERC_OTS_PSL02 |
Do you have the capacity to rollback to previous versions of your product? |
ORC_ERC_OTS_PSL03 |
Provide details of any associated processes / procedures and tools that are used. |
ORC_ERC_OTS_PSL04 |
Are the processes for accepting and responding to technical faults from end users appropriate? |
ORC_ERC_OTS_PSL05 |
Do you provide online support for user queries? |
ORC_ERC_OTS_PSL06 |
Do you proactively monitor running of systems and system components to automatically identify faults and technical issues? |
ORC_ERC_OTS_PSL07 |
Provide details of any associated processes / procedures and tools that are used. |
ORC_ERC_OTS_PSL08 |
Do you have a documented roadmap for future development of your product? |
ORC_ERC_OTS_PSL09 |
Provide details of planned development, technical updates. |
ORC_ERC_OTS_PSL10 |
Does the Developer provide details of how they will ensure the continued availability of their product? |
ORC_ERC_OTS_PSL11 |
Do you have a plan for decommissioning your product? |
ORC_ERC_OTS_PSL12 |
Describe your processes for decommissioning your product and dealing with any identifiable data. |
ORC_ERC_OTS_PSL13 |
Do you have a plan for dealing with any identifiable data in the event that an individual stops using your product? For example by installing or unsubscribing. |
ORC_ERC_OTS_PSL14 |
Provide details of any associated processes / procedures and tools that are used. |
ORC_ERC_OTS_PSL15 |
Does the organisation follow any formal testing standards |
ORC_ERC_OTS_PSL16 |
Provide details of any associated processes / procedures and tools that are used. |
ORC_ERC_OTS_PSL17 |
For each of the following if they are carried out please describe the people / roles that are involved, the processes that they work to even if they are informal. |
ORC_ERC_OTS_PSL18 |
Unit |
ORC_ERC_OTS_PSL19 |
Regression |
ORC_ERC_OTS_PSL20 |
End-to-end / Integration |
ORC_ERC_OTS_PSL21 |
User Acceptance |
ORC_ERC_OTS_PSL22 |
A/B |
ORC_ERC_OTS_PSL23 |
PEN / Vulnerability |
ORC_ERC_OTS_PSL24 |
Testing across devices |
ORC_ERC_OTS_PSL25 |
Load / Performance |
ORC_ERC_OTS_PSL26 |
Security |
ORC_ERC_OTS_PSL27 |
Other non-functional tests |
ORC_ERC_OTS_PSL28 |
Other testing |
ORC_ERC_OTS_PSL29 |
Has the Developer provided sufficient evidence to satisfy all the requirements of the product's testing? |
ORC_ERC_OTS_PSL30 |
Technical Security
Question |
Question Reference Source |
Does the organisation have ISO27001:2013 accreditation? |
ORC_ERC_SEC_ORG1 |
Is the certification body in the UKAS list of ISO27001:2013 certification bodies? |
ORC_ERC_SEC_ORG2 |
Please provide Statement of Applicability? |
ORC_ERC_SEC_ORG3 |
Does the scope include product and associated services? |
ORC_ERC_SEC_ORG4 |
Is the Application a Native Application for a Mobile Device? |
ORC_ERC_SEC01 |
Is the Application a Web Application? |
ORC_ERC_SEC02 |
Are Web API’s accessed? |
ORC_ERC_SEC03 |
Does the App access, process or store Personal and/or Sensitive Data? |
ORC_ERC_SEC04 |
Is sensitive data persisted to the mobile device? |
ORC_ERC_SEC05 |
What Permissions does the Application request? |
ORC_ERC_SEC06 |
OWASP Level is required in order to review evidence. IF Mobile = Y IF 'Personal and /or Sensitive Data is accessed, processed or stored’ = Y OWASP Level then MASVS = 2 IF Sensitive data is persisted to the device then MASVS = 2+R ELSE OWASP Level then MASVS = 1 IF Web = Y IF 'Personal and /or Sensitive Data is accessed, processed or stored' = Y OWASP Level then ASVS= 2 ELSE OWASP Level then ASVS= 1 |
ORC_ERC_SEC07 |
Does the Application connect to a Medical device? |
ORC_ERC_SEC08 |
Does the Application connect to a healthcare service? |
ORC_ERC_SEC09 |
Does the Application provide Alerts or Notifications? |
ORC_ERC_SEC10 |
Does the Application provide Suggestions? |
ORC_ERC_SEC11 |
Does the Application undertake Calculations? |
ORC_ERC_SEC12 |
Does the Application support in-App purchases? |
ORC_ERC_SEC13 |
Has a Security Assessment been undertaken by an accredited external third-party? |
ORC_ERC_SEC14 |
Is the external third-party a CREST / APMG / CHECK registered supplier? |
ORC_ERC_SEC15 |
Does the scope of the report cover the full Technical Architecture of Application? |
ORC_ERC_SEC16 |
Has an industry-standard been used for the risk model in the associated PEN/Vulnerability testing? |
ORC_ERC_SEC17 |
Have all ‘Medium’ Risks / Issues identified been mitigated and resolved; and can this be demonstrated through retesting within six weeks from the original PEN / Vulnerability testing? |
ORC_ERC_SEC18 |
Has the Code-Level Security Assessment been undertaken against the correct OWASP Level? |
ORC_ERC_SEC19 |
Is the methodology for the Security Review proportional to the attack service and risk of the Application? |
ORC_ERC_SEC20 |
The project is run by N!P, jointly funded by Nordic Innovation and Nordic healthtech industry and powered by ORCHA.