Asset 32@1.5x-1

Whether you're looking for a complete exam development solution or just want to increase efficiency, we can help.

The Certiverse platform is modular, making it easy for experienced test sponsors to plug into the system at whatever stage of the development cycle is needed. However, organizations that are looking to develop an entire exam—and deliver it—will experience the maximum benefit of partnering with Certiverse.

We include all the steps that traditional exam programs use to ensure high quality. The difference is that we’ve baked these into an automated system that streamlines processes and virtually eliminates between-process delays. In other words, we didn’t cut corners, we just straightened the road.

Certiverse's transformative approach to exam development 

END-TO-END INNOVATION

Certiverse follows all the steps of the traditional test development process but in a revolutionary way. We use AI, machine learning, and natural language processing to automate workflows and provide intelligent prompts to guide users. Our technology reduces the need for training and facilitation and accelerates speed-to-market. 

Compare our processes to discover what makes Certiverse different

The job task analysis (JTA) is a multi-step process to establish a set of domains and associated knowledge and skills deemed necessary to competently handle the responsibilities of a particular job or role. The first step is to identify and rank the essential and critical tasks for someone who is proficient in that job/role. This is accomplished via input from subject matter experts (SMEs) who document the necessary tasks, and is then followed by a survey that yields information to help organize and weight them. 
How its traditionally done The Certiverse Model

The JTA is done in person or by web conference. Traditionally, 10 to 12 subject matter experts (SMEs) travel to a location where a facilitator provides training and guides participants through the workshop. These typically last 3-5 days.

Certiverse enables the facilitation of an asynchronous JTA, so  participants can work when it is most convenient for them. SMEs are invited to participate through system-generated email and can be anywhere in the world with an internet connection. Video instruction and built-in training are provided through the platform, so your staff can manage the process. Certiverse uses natural language processing to support the writing of various tasks. The innovative platform assists SMEs in easily organizing the tasks into domains, so that the output of the JTA is a rough draft of the exam blueprint.

Timeline: SME availability and scheduling can affect the timeline.

Timeline: No scheduling is needed. The program sets the window of time for completion of the JTA, and participants log in to complete the work independently, when it’s most convenient. Since participation doesn’t interfere with job or family commitments, this enables a broader pool of SMEs to participate.  

Cost: Varies by program. Many SMEs are compensated for their time, and all travel expenses (airfare, hotel, meals, etc.) are covered by the credentialing organization. Many programs hire an outside facilitator to conduct the workshop, if they do not have in- house expertise.

Cost: No need for an outside facilitator. Functionality is included in SaaS pricing or may be purchased as a standalone service.

Considerations: Aside from being challenging to schedule, requiring SME travel inherently rules out certain groups who may not be able to attend an extended meeting due to job or family commitments, disability or geographical constraints. Utilizing only those SMEs who can travel or attend a scheduled web conference may introduce unwanted bias into the exam. Also, there’s a risk that some SMEs may dominate or monopolize the conversation during a group meeting, discouraging input from quieter group members. Lastly, requiring group travel contributes to excess carbon emissions and may hamper an organization’s ability to meet corporate goals around reducing its carbon footprint.

Considerations: While the asynchronous aspect of Certiverse enables programs to enlist a more diverse pool of SMEs and appeals to those individuals who are more introverted, some contributors like the camaraderie and networking involved in a traditional JTA workshop.

In addition, gamification within the platform incentivizes SMEs by providing points for items that are written, reviewed and included in the final exam.

 

After the job tasks are documented, participants are sent a survey that asks questions about the identified responsibilities and competencies to help determine the relative importance and weighting that each should receive in the resulting exam.

How its traditionally done The Certiverse Model

The survey is generally conducted using a tool such as Qualtrics or SurveyMonkey, with the program sending participants a link to the survey.

AI and machine learning are used to compile the information from the JTA into a survey. Certiverse then automatically publishes and sends a survey link to participant emails provided by the client.

Timeline: The timeline for responding to the survey is determined by the program and may require that follow-up reminder emails be sent to participants.

Timeline: The timeline for responding to the survey is determined by the program. Follow-up notifications are automatically generated from the system to remind SMEs to complete the survey promptly.

Cost: Depends on the tool used and whether a consultant’s guidance is needed to create the survey.

Cost: Included in the subscription and in the Pay-As-You-Go per-blueprint fee.

Considerations: It’s important to understand how to construct a good survey based off the information gathered in the JTA or the resulting data will not be actionable. For some organizations, this may require outside consultation. 

 

Data from the JTA survey results are analyzed from a psychometric perspective to determine the criticality score of each identified task within the job role.

How its traditionally done The Certiverse Model

Response data from the survey system is exported and imported into analysis software. A psychometrician writes syntax to perform analyses on the data.

Built with the oversight and input of Certiverse’s chief psychometrician, the platform automatically analyzes survey data giving a criticality score to each task.

Timeline: This step is time-consuming as much of the analysis is a manual process. An in-house psychometric team will be able to perform these tasks more quickly than a psychometric consultant who may not be as familiar with the program.

Timeline: Because this step is automated, it can be done in a matter of minutes, instead of weeks.

Cost: Varies based on staffing/salaries. For organizations lacking in-house psychometric expertise, this step may be particularly costly.

Cost: Included in Certiverse pricing.

Considerations: To qualify as a psychometrician, an individual generally has a PhD in statistical analysis or equivalent education and experience, so individuals possessing this qualification may be scarce.

 

The exam blueprint is a detailed outline of what will be covered on the certification exam. It not only captures the necessary topics that need to be covered, but also the level of depth for those topic areas. This blueprint is used by the item writers to create test questions that will allow the credentialing organization to accurately measure competency.

How its traditionally done The Certiverse Model

Oftentimes this is done via “blueprint-by-committee” with a lengthy in-person or web conference involving the test sponsor, psychometrician and SMEs. Alternatively, the blueprint may be created with an iterative process involving asynchronous editing of a shared document (e.g., Google Docs) to incorporate various stakeholder edits and comments.

The exam blueprint is automatically generated from the survey data and includes item-writing assignments based on the weightings of various subject areas.

Timeline: Based on the schedules/availability of participants.

Timeline: The exam blueprint is created in an instant.

Cost: Varies due to time commitment of those involved.

Cost: Included in the subscription. Pay-As-You-Go customers pay a per-blueprint fee.

The heart of the exam development process, item writing is the process of creating test questions and response choices. A good question has many elements: it should be clearly stated and understandable, it should be free from bias and not require specialized knowledge outside of that which is being assessed, it should be legally defensible, and the answer choices should be definitive and unambiguous.

How its traditionally done The Certiverse Model

This is done via a web conference or in-person workshop with a facilitator managing the process and providing detailed instruction to SMEs about the construction of a strong item. During the item-writing and review steps, considerable time is spent managing the process—following up with SMEs, managing the review and approval process, and tracking completed items against the number needed per domain.

Certiverse facilitates asynchronous item writing by using natural language processing and machine learning to coach and guide item contributors into authoring strong items.

Timeline: Based on the schedules/availability of participants.

Timeline: Participants can work on items in small fragments of time or in longer sessions (as their schedule allows). Certiverse makes it easy for contributors to pick up where they left off and the system provides clear action items on remaining steps. Once an individual item is written, the system immediately initiates the review process for that item.

Cost: Varies due to time commitment of those involved.

Cost: Included in the subscription. Pay-As-You-Go customers pay a per-item fee.

Once they are written, items must be reviewed for accuracy. The best people to do this task are the other subject matter experts working on the exam. In addition to reviewing for technical accuracy, items should also be reviewed for spelling/grammar, clarity and to ensure lack of bias.

How its traditionally done The Certiverse Model

This is done via a web conference or in-person workshop with items receiving multiple reviews and revisions. The process is led by a facilitator who ensures all items are reviewed from an editorial perspective for consistent tone, grammar, etc.

As items are written, Certiverse assigns them to other SMEs for review. A chat function in the platform allows reviewers to ask clarifying questions of the item writer.

Timeline: Based on the schedules/availability of participants.

Timeline: All reviews and revisions happen asynchronously. The system keeps track of the various stages of each item, so contributors know what work needs to be completed each time they log in.

Cost: Varies due to time commitment of those involved.

Cost: Credential programs can utilize Certiverse’s unique revenue-sharing model to incentivize SMEs; paying them for each approved item or with a royalty program that compensates them each time their item is presented in a live test. This not only incentivizes SMEs to produce high-quality items, it also allows credentialing organizations to create an exam with minimal upfront costs.

Considerations: Because this activity is often done as a group, the item pool may be exposed to all of the SMEs, introducing a security risk for some programs.

Considerations: Reviewers only see the items assigned to them, greatly reducing exposure and opportunities for item theft.

Like a test-run of the actual certification exam, the beta exam is given to test takers who were not involved with the development process. Individuals agree to be beta testers in exchange for early access to the test and sometimes receive a free or discounted exam. The beta test gives the exam sponsor an opportunity to troubleshoot any problematic items before releasing the final certification test to the general public. Beta testers are informed that they will not receive a pass/fail notification until after a sufficient number of test takers have completed the exam, and the data has been analyzed.

How its traditionally done The Certiverse Model

For many organizations, test assembly and the publishing of test forms is a time-consuming process, so some skip this step because of the work and time involved.

Certiverse publishes beta exams through our own test driver.

Timeline: The timeline is contingent on many variables including test assembly and publishing timeframes as well as beta tester recruitment. Advance marketing of the certification is critical for identifying potential beta testers.

Timeline: Test assembly and publishing can be completed quickly. However, it’s still critical that credentialing organizations pre-market the certification to identify potential beta testers. The more quickly test takers are identified and able to sit for the exam, the more quickly the exam can move on to the final stages of development.

Cost: Varies based on the exam driver used and related pricing for test assembly and form publishing.

Cost: Included in the subscription.

Considerations: Although it may be tempting for some programs to skip this step, beta testing is a critical part of the exam development process and yields essential data necessary for accurate standard setting.

 

The beta test gives the exam sponsor an opportunity to troubleshoot any problematic items before releasing the final certification test to the general public. Response data related to each item in the beta test is analyzed to determine whether the test question performed as intended.

How its traditionally done The Certiverse Model

Response data from the exam system is exported and imported into analysis software. A psychometrician writes syntax to perform analyses on the data.

We provide psychometric item analyses in a simple, easy-to-understand report.

Timeline: Much of this analysis is a manual process. An in-house psychometric team will be able to review response data more quickly than a psychometric consultant who may not be as familiar with the program.

Timeline: The item analyses report of the beta exam is available after the conclusion of beta testing, so that recommendations can be implemented  and standard setting can begin.

Cost: Varies based on staffing/salaries and whether the organization has in-house psychometric expertise.

Cost: Included in the subscription.

A standard-setting panel gathers to evaluate data (including the results of the beta exam, if there was one) in a systematic way for standard setting to determine an appropriate and fair cut score for the exam.

How its traditionally done The Certiverse Model

This is done via a web conference or in person with SMEs following a method outlined by senior test development staff. Often, there are policies that are followed (for example, the panel making a recommendation to the board who then set the final standard), but these vary considerably across exam programs. Preparing for and leading the meetings may be labor intensive, preparing paper forms, entering and analyzing data during the meeting, etc. 

Because every program has unique needs, Certiverse is flexible. We can replicate the traditional standard-setting approach, if required. However, we are currently developing a module that will streamline the process so it consists of two short virtual meetings with the bulk of the panel's work being done asynchronously via the platform to make secure, individual decisions. And lastly, for programs that want the quickest results, we have designed a rigorous process that can take place in a single, 90-minute virtual meeting. 

Timeline: Based on the schedules/availability of participants, often with considerable lead times required as well as post-meeting activities.

Timeline: This may take from one day to multiple days, depending on the standard-setting approach chosen by the test sponsor.  

Cost: Varies due to time commitment of those involved.

Cost: Certiverse's streamlined virtual standard setting is included in the subscription. Further consulting related to the standard-setting process is available for an additional fee. 

Test assembly is the process of selecting individual items to create a valid test form. For security reasons, exam sponsors typically create several exam forms and test assembly must ensure that those forms are all valid with equivalent standards. 

How its traditionally done The Certiverse Model

A test developer exports item meta data from the exam system and uses a combination of analysis software, spreadsheets, and paper forms to select appropriate items. The psychometricians perform equating; wherein they apply the standard that was determined during the standard-setting process to set the appropriate cut-score for each form. In some cases, an independent equating study must be done for each new form.

Certiverse produces content-valid forms and applies the equating process to create a scoring table, with optional review by SMEs also available.

Timeline: This step is time-consuming when done manually, as input may be needed from other individuals whose schedules may vary.

Timeline: Test assembly can be completed in a few hours per form. This functionality is currently being enhanced to handle output to multiple forms more quickly and automate the equating process. Optional SME review may vary depending on exam structure and length.

Cost: Manual test assembly requires staff time, although test assembly can be outsourced for a fee.

Cost: Included in the subscription. 

Test publishing is the process of publishing a single test form that pulls from a larger item bank to create countless variable exams (ensuring that each candidate sees a unique combination of individual items that, together, accurately measure all of the competency areas identified on the exam blueprint).

How its traditionally done The Certiverse Model

This process is generally done through a handful legacy delivery systems that publish exams to be taken and proctored in test centers or remotely.

Certiverse automatically publishes exams through our own test driver, which can then deliver exams to test center or remote locations.

Timeline: This process can take anywhere from 8-12 weeks or longer.

Timeline: This process takes minutes, not weeks.

Cost: Varies based on delivery provider and method.

Cost: Test publishing is included in the subscription. Pay-As-You-Go customers pay a separate per-delivery fee. This delivery fee is significantly discounted for SaaS customers.

Considerations: Some delivery systems are built for one delivery method (test center or remote) whereas some can publish to both formats. Test sponsors should choose a system that is compatible with their overall delivery strategy.

 

 

Security and forensics must be considered at every stage of the exam development process. If exam questions are easily stolen or if candidates can otherwise cheat to get a passing score on your test, it will no longer be a reliable indicator of competency and your credential will have no value among employers or job seekers in the market.

How its traditionally done The Certiverse Model

Tests are delivered in proctored test centers to deter item theft. Data forensics are run after the exam is completed to detect any anomalies while testing and to assess whether individual items are likely to have been exposed.    

A key security strategy that Certiverse employs is enabling test sponsors to create larger item banks and to revise exam content more frequently. This is possible because we automate many facets of the process that typically require a significant investment of time and personnel.

Timeline: The delivery vendor may not report incidents until days or weeks after they occur, thereby increasing exposure.

Timeline: At a rate of approximately one hour of SME time per item, new items and test forms can be created rapidly, as needed, to mitigate the threat of item exposure. A dozen SMEs can create a new 50-item form in just a few days. 

A future development goal is to incorporate real-time analysis to detect cheating patterns and combine that with “light” live proctoring or AI-based proctoring (e.g., “record and review”).

Cost: There are several factors that factor into the cost of security—depending on the choices a program makes, savings in one area might be offset by security exposure in another. Credential management systems with psychometrics capabilities can be expensive.

Cost: Certiverse lowers the overall cost of item-creation, so test sponsors are less reliant on test center delivery or expensive proctoring solutions as a means to protect valuable content. 

 

Considerations: Factors that influence security and exam integrity include:

  • Test center vs. remote-proctored delivery
  • Live proctoring vs. record-and-review
  • The size of the item bank and standard age of items
  • Test development expenses and frequency
  • Forensic and security know-how (internal or outsourced)

 

Considerations: Lowering exam costs and expanding delivery options increases the accessibility of your exam without compromising quality.

Reach out today to discuss how Certiverse can help you launch something new. 

CONTACT US