Real Time Scoring


Once the user trained a Machine Learning model, KPI generator allows to evaluate the quality and the performance of the model.


  • There are two types of KPIs. The first type of KPIs are calculated based on gold annotations and help quantify the correctness of the answers given by the model.
    For example:

    • Precision and recall
    • Receiver operating characteristic curve and the area under it.
    • Confusion matrix

    are all quality-related KPIs. For users with more in-depth technical knowledge, it is possible to observe the optimization process of your model. For example, how is the loss curve decreasing ? You can also turn-on the ability of the Eval engine to constantly track quality drifts and notify you when appropriate.

  • The other types of KPIs do not require gold annotations. They quantify the quality of the interaction between you, the user, and the model. For example:

    • How long the model takes to calculate its predictions
    • How many answers did the model make a) in total, b) and for each label on a given collection of documents

    are all performance-related KPIs.

  • Another feature of the Eval engine is the automatic calculation of statistics on the annotated dataset you configured via 2OS platform. As soon as you finish creating it, statistics such as:

    • the number of tags
    • the number of annotations labelled with each tag
    • the number of documents
    • the average number of pages in your documents
    • and a lot moreā€¦

    are calculated and nicely put into diagrams that you can pin to a documentation or presentation slides.

How it works

KPI Generator

Figure 1: Pipeline: KPI GeneratorKPI Generator


  • You get to take the relevant decision when your model quality lowers. Should you annotate more data ?

  • Allow your model to be optimized longer

  • Set up the parameter adequately

Use Case

After training an AI model that detects useful information to you such as persons names, or other specific names, use the Eval engine to know how well your model is doing both in terms of quality and performance. Even if you forget to check these metrics yourself, the Eval engine will notify you if anything goes wrong and suggests to you the steps that you may consider to take to put your model on track again.