Introduction

ORCHA’s mission is to get more people using Digital Health solutions and making this exciting new frontier in healthcare part of mainstream health and care delivery. A key tool in delivering this is the ORCHA Review Process which allows us to analyse and assess a Digital Health product or service to ensure that it meets all the relevant regulatory standards and guidance that relate to it and to gather and garner a wide array of additional insights and data that can support a range of downstream requirements such as Accreditation and ‘App’ matching. 

Accreditation of Digital Health solutions is a fundamental foundation for their wider adoption and integration into health and care settings. It is clear that without a trusted accreditation approach, health and care organisations (and the professionals within them) simply can not develop the confidence needed to start to embed these novel products into their day to day practice and processes[1].

Digital Health Accreditation however poses major challenges to all health systems globally. Whilst most advanced jurisdictions recognise the advantages that Digital Health can offer, most (if not all) existing technology accreditation or evaluation models and approaches are simply not up to the unique challenges that the world of mobile Apps and digital services present.

The key accreditation challenges are:

• Achieving sufficient assessment rigor in a cost effective and pragmatic manner

• Balancing the level of rigor with the need to maintain product and service owner (Developer) engagement

• Managing the rapid change cycle that most (good) digital health solutions adopt – and the associated re-assessment burden

• Managing the requirement to regularly update accreditation models as new regulations, guidance and technologies emerge – and the associated re-assessment burden

• Delivering sufficient volume of accredited solutions from a standing start to support local demand/opportunity in a cost effective manner

• Funding a new system of governance and accreditation and sharing these costs equitably across the system.

• The number of different solutions that are now available are in the hundreds of thousands and traditional models are not designed to cope with this volume of distinct products reliant as they almost universally are on an expert-based evaluation approach. 

ORCHA’s Review methodology is one of the most advanced approaches to Health Technology assessment available and uniquely combines cutting edge technology with human evaluation and analysis.

Our Reviews consist of two levels of analysis:

• the ORCHA Baseline (“OBR”); and

• the ORCHA Enhanced Review Components (“ERC’s”).

ORCHA provides a unique solution to all these challenges many of which are founded on the layering principle outlined below.

The OBR is a ‘desktop’ assessment of a digital health solution from publicly available information.

The OBR process can be used by health and care systems, commissioners and providers to identify solutions that have the potential to support specific aspects of their service or particular cohorts of their populations. 

The ERC’s build on the baseline process and investigates the solution in more depth in all of the ORCHA Review Domains (detailed below)|. Many ERC’s require engagement from and with solution owners/developers as it looks at aspects of the solution that are not normally easily assessed using only information in the public domain. The ERC typically involves the collection of additional information and evidence from owners/developers that support a full compliance analysis in regards to relevant regulations and standards and a more detailed assessment of adherence to best practice and local guidelines and policies.  

The combination of the OBR and relevant ERC’s can support a wide variety of Accreditation regimes and can accommodate local rules and regulations and individual pass/fail thresholds, enabling national, regional or local accrediting bodies to set their own rules and standards depending on the Accreditation context.

This already forms the basis of our Digital Health Assessment Support for NHS Digital, the Dutch national mental health body and a growing number of Ministries of Health around the world and are all part of the ORCHA Accreditation Community which delivers considerable efficiency into the process of Digital Health Accreditation through the sharing of assessment burden. 

In addition to the ‘compliance data’ captured through the OBR and ERC’s we also capture a huge amount of ‘characteristic’ data about the Digital Health solution. This data goes beyond ‘compliance’ or quality assessment and includes information about key features and functions of a solution, insights into who the solution is aimed at or designed for, (or popular with), and data on what other products/services or systems the solution ‘works with’ or interoperates with. This ‘characteristics’ data is key to supporting the ORCHA App Finder which is a sophisticated search engine specifically for health and care Digital Health solutions and which incorporates our unique App Matching features that allow a user to tailor their search to find the solution that works best for them. The ORCHA App Finder drives all our search functionality across all ORCHA Libraries, Catalogues and Formularies.

Our Review Process is able to assess Apps and Web-based solutions alike. We use the term App as a generic description of these products and services and many such services exist in both App and Web forms. 

The Review Engine

A key and relatively unique feature of the overall ORCHA approach to assessment, is that our assessments are dynamic and responsive to the type and nature of the solution under review. All Apps and related solutions are not the same and there are a huge array of differing features, functions and areas of focus across the c360k products that are currently available globally. It is therefore crucial that any assessment approach can be responsive to the specific characteristics of a solution, whilst at the same time ensuring that this is always consistent and transparent and that like for like solutions are always treated the same way.

One example of this is in the area of regulation. The number of regulations that an App falls under the scope of, can differ markedly, with relatively simple ‘wellbeing’ focused Apps triggering relatively few regulatory burdens, versus a sophisticated, condition focused App, that is likely to trigger significantly more. It is crucial that an assessment approach can cater effectively for this variation. This is a major challenge in this complex arena and the ORCHA Review Engine is a crucial tool in delivering this for both the OBR and for our enhanced reviews and accreditations. 

The ORCHA Review Engine has been designed to support all the elements required to deliver a robust and effective assessment process for Digital Health solutions. It fully automates an assessment workflow, supports role based Assessor inputs and queues and manages multiple scoring or decision mechanisms. It can be used independently of the ORCHA Review team or in combination, with Client Assessors taking responsibility for specific areas of a Review.

The Engine manages Assessment or Review queues from the ‘raw’ data extracted from the various App stores or web solutions repositories, through to the publication via a range of API’s of Review or Assessment outputs onto relevant ‘Libraries/Stores, Catalogues and Formularies’ which can be driven by the App Finder solutions or created and operated independently. 

The Review Engine offers a range of automated Assessment techniques and processes built around the significant layers of data it holds on all the Digital Health solutions in its repository. This includes the innovative use of AI technologies to auto assess a number of areas where large data sets offer significant insights into many aspects of a Digital Health solutions performance and impact. 

What ORCHA has set out to deliver in this space is a flexible and highly effective approach and technology tool set, to delivering all of these assessment layers and dynamic variants, to support whatever end goal an organisation or body has and to build on each assessment layer we deliver to avoid the need for rework or duplicated analysis.

Ultimately the ORCHA Review Engine fuses human expert analysis with technological and data driven integration methods, to create what we believe is the most advanced health care technology assessment solution available.

The Review Team

The ORCHA Review Team is a unique collation of Digital Health expertise. It combines dedicated professional Digital Health Analysts with a wider group of Subject Matter Experts who cover all aspects of Digital Health compliance and analysis. 

The ORCHA Digital Health Analysts or ‘Reviewers’ are all professional Digital Health assessors who are trained specifically to deliver reviews and assessments in this specialist environment.

All are graduate level or post graduate level and they undertake a highly structured and ongoing training programme through our ORCHA Digital Health Assessment Academy which builds up their skills and expertise in all the key review domains.  This is reflected in a ‘Belt’ system with Reviewers progressing through the various colours and levels of belts as there experience and expertise grows. After an initial period of training, the Reviewers start to specialise in a specific area of Digital Health and become the Review Leads for these areas. This enables them to form a very clear view of the solutions available in a specific condition or pathway. Through this process, the Lead Reviewers will engage extensively with the Developers of the solutions in their focus area and will build relationships with these organisations through the delivery of the more advanced accreditation processes. The Lead Reviewers also work closely with their associated Lead Clinicians or Professionals and this forms a powerful combination of insight and expertise in any given area. 

This Reviewers are augmented by a wider network of ‘experts’ in specific areas of Digital Health, such as our clinical advisory group that includes health and care professionals from a wide array of disciplines who take on the role of Clinical/Professional Leads for certain conditions and/or pathways. We also have data and security experts who are often leading the discussions internationally around Digital Health compliance in these areas. We have experts in Evidence evaluation and Legal and Regulatory issues as well as commercial and financial matters.

In addition to the expert advisory groups we also have a growing array of specialist partners that also support specific areas of the Review and in particular support the delivery of some of the Enhanced Review Components outlined below. 

Clients can take advantage of our ORCHA Digital Health Assessment Academy to support the development of their own Client Assessors. 

Monitoring and Surveillance 

Market Surveillance

A crucial aspect of any accreditation delivery process is the ongoing management of approved solution change. 

All our accreditation and review services incorporate the ‘closed loop’ principle, which ensures that any reviewed or accredited product or service is constantly monitored for changes and in some instances ongoing compliance and any such changes are immediately flagged on any related ‘front end’ manifestation of the accreditation or review output. The closed loop principle also supports the management of re-review or re-accreditation processes that result from material changes in a given product or service. 

The closed loop is delivered through the features and functions built into the Review Engine. Crucially this incorporates a monitoring and surveillance solution that can detect changes in relevant approved solutions using a combination of data scraping and natural language processing. This is augmented by Developer updates that are provided directly into the platform via the Developer Compliance Vaults. 

This unique monitoring solution ensures that material changes to approved solutions are detected and appropriately actioned in an automated way. Typically this involves all ‘front end’ libraries, catalogues or stores being updated with a warning flag to indicate the solution in question has changed and is being re-assessed. In the back-end the detection of a change will always initiate an OBR ‘re-review’ within a 14 day period. This is usually used as the trigger for any ERC ‘re-reviews’ that might be required as a result of the change. All of these elements are configured in the Review Engine to reflect the specific approach an individual client wishes to take.

The closed loop model also allows for the monitoring of approved solutions that have failed to deliver an update within a prescribed time frame. For the OBR this is 18 months and any solution that has not had a basic update within that timeframe would typically not be assessed and if it had been previously assessed and subsequently became out of date, its score would typically degrade up until 24 months from the last update at which point it would be zero rated. Most accrediting bodies follow this rule although some require even more regular updates and this can be adjusted depending on the nature of the solution and the risk associated with its use.

The Review Engine enables:

• A weekly monitoring of the Apps available in the relevant sections of the iTunes and Google Play store plus selected websites and web based solutions to detect changes via new version numbers, update dates and notifications from the developers

• The automatic flagging of a new version or update on all published reviews of that solution

• An automatic alert of a new version to relevant Review Owners/Admin for any other reviews of that solution (via the Review Engine and/or email)

• The automatic allocation of any new versions or updates into an OBR Re-Review queue and if required any other Review/Accreditation Re-Review queue

• An initial inspection of any updated Apps to identify the materiality of any changes (undertaken as part of the OBR cycle)

• A full re-review/re-assessment of the relevant solution for the OBR within 14 days of the new version alert (formal publication of any updated review will be within 24 days to allow for the 10 day Developer Feedback period) 

Queue Creation and Market Curation

One of the key features of the Review Engine as noted above, is the automated monitoring of the Digital Health market place.  Through a series of automated stages - on a weekly basis – we assesse and reassess the Digital Health ‘App’ market as represented by the Apple iTunes App Store and Googles Play Store. We also have developed a methodology for monitoring identified web only Digital Health solutions that has been built into this process. Having collected the market data we then undertake a number of automated ‘enrichment’ processes which includes:

• Automatically categorising the Apps across over 250 health conditions and categories

• Auto-Coupling an Android App with its iOS counterpart

• Undertaking an initial Automated Indicative Assessment (in development)

For the OBR process, at this stage, we exclude from the solutions Backlog any Digital Health solutions that haven’t been updated in the prior 18 months and we also exclude any Apps that are not available in ORCHA’s currently supported jurisdictions.  

These two primary exclusion criteria, filter out around 65% of all the available solutions and leaves a still sizeable but much more addressable ‘Viable ‘App’ Backlog’ from which we form our various review ‘queues’. 

The rational for excluding solutions that have not been update within the last 18 months is multi-levelled. Whilst there are some Apps that are successfully used without updates within this timeframe, they are the exceptions not the rule. In general terms, it is considered good practice to update Apps as a minimum whenever there is a major update to the Android or iOS operating systems. In addition, one of the best (and most challenging) features of Apps is their ability to adapt over time and to continuously improve and iterate their user interfaces, user experience and functions and features. In the critical health and care space, updates are also a crucial indicator that any relevant clinical or medical information or guidance is being kept relevant and in line with current best practice and available knowledge.  

Once the exclusion filters have been applied, we organise our Viable App Backlog by category area and within each category area we organise the Apps by the most downloaded and within these bands the most recently updated. It is worth noting that only the Google Play platform provides download data. In order to manage the lack of this data for iOS Apps, the Auto-Coupling process ensures that an iOS App is always queued with its Android companion. This doesn’t resolve the issue for iOS only Apps. These form a separate queue ordered by most recent updates.

Our standard proactive review cycle then takes an App from each category in turn which ensures that we review the full breadth of Apps available rather than simply focusing on only the most downloaded which would in practice mean predominantly Apps in the diet and fitness arena. 

The ORCHA Baseline Review

The ORCHA Baseline Review (“OBR”) is ORCHA’s first level of assessment and involves a detailed ‘desktop’ analysis of Digital Health solutions looking across all of the key areas of regulation and compliance. The OBR is largely undertaken proactively as part of ORCHA’s ongoing assessment and monitoring of the whole Digital Health market place and we review the most downloaded and most recently updated Apps and related Digital Health solutions across over 250 health and care categories and conditions. To date we have reviewed over 6,000 such products and whilst we continue to review more than 500 new solutions a month we also undertake re-reviews of all updates and new versions of the solutions we have already looked at. Full details of the OBR can be found at V6 Documentation Version 2

Enhanced Reviews 

After the OBR process has been completed and the results are published, the Enhanced Review stage can commence. This can be triggered by a number of things including: 

• A Client request to undertake relevant ERC’s for the purposes of a specific accreditation process – this will typically be as a result of the Client identifying the solution in question and proactively adding it to their ‘backlog’ or as a result of a Developer application to participate in a relevant accreditation process that the Client accepts;

• A Pro-active ORCHA review – which will for some ERC’s be undertaken automatically by ORCHA if an app meets certain characteristics

• A Developer requested review where a Developer wishes to understand their compliance with specific CRC’s or to use the CRC data to support their wider operations. Developers can also use this route as a mechanism for ‘opting in’ to the various accreditation processes managed by ORCHA within the Review Engine. 

Most ERC’s involve additional human analysis by our Review Team. Some however are the result of the collation by ORCHA of a range data sources that we have combined and analysed to create unique insights in certain areas. The current menu of ERC’s is summarized below:

We are consistently adding ERC’s as we evolve and new assessment opportunities emerge.

Once an ERC assessment has been undertaken – and subject to any subsequent updates or changes to the solution not triggering a re-assessment/re-review – this is available to all Clients who have requested that ERC as part of their accreditation process. This not only enables ORCHA to make the overall process of accreditation hugely efficient, reducing considerable duplication of effort across the ORCHA Accreditation Community, but also allows us to drive more solutions into various accreditation processes through highlighting to the Developer an apps ‘eligibility’ for these based on the ERC’s already undertaken. For smaller jurisdictions this can aid considerably with the number of solutions engaging with any given process.

Review/Accreditation Key Roles

There are a number of key constituencies in a typical process, including:

• The Applicants – the Developers who are applying to achieve the accreditation or approval

• The Assessors – the individuals’ tasked with undertaking aspects of the assessment process

• The Administrators – the individuals’ tasked with managing the progress of all applicants and assigned tasks to Assessors

• The Owners – the individuals’, bodies or boards that oversee and own the overall process

Each of these groups require different things from the systems that support the process and automating the workflow as much as possible between each of these key players is a key part in delivering efficiency into the overall model. The Review Engine component supports all of these groups roles within an accreditation process.

Assessors 

Assessors have their own Accounts and Login’s and within these their own queues, portfolios and data insights. Their involvement and inputs are managed via alerts and notifications and the Review Engine provides them with a single interface for all their assessment tasks, including guidance notes, support structures and the assessment processes themselves.

Applicants 

Applicants are provided with an application portal where they can initiate or confirm their involvement with a relevant accreditation process. This is usually linked to the ORCHA Developer Account and allows for relevant Developers within the ORCHA Developer Community to be notified of opportunities to engage with a new accreditation process. The Applicant Portal also allows Developers to provide additional information to support aspects of the accreditation (ERC or CRC elements for example) and also provides Developers with continuous status updates, alerts and notifications throughout the process. The Applicant Portal ultimately will manage and deal with successful and unsuccessful Applicant communications and associated processes, such as the delivery of Improvement Reports or Accreditation Agreements.

Administrators 

Administrators are able to view all aspects of the accreditation process including the status of ‘Live’ accreditations and the assessment backlog. This will show them where every solution within the process is at and who is responsible for it. The Administrator Account also allows for Assessors to be managed, added, changed or for specific solutions to be assigned or re-assigned to specific Assessors. Where ORCHA Reviewers are part of the process, Administrators will also see the status of tasks with the ORCHA Review Team and they can send messages and queries within system around this. The Admin suite also includes dashboards and reports detailing various aspects of the accreditation process.

Owners 

Accreditation Owner privileges mimic the Administrators but in addition allow for limited editing and new accreditation version development and the management of Administrator Accounts.