Troubleshoot the safetynet

TQL is versatile tool that can help you maintain and improve your system. In this section we'll walk through some strategies for exposing weak points in a bot's repertoire

View inputs to the safetynet

The query to pull out user inputs that hit the safetynet could look like this:

lu t.e1.userInput: t.e2.fname ~= ".*[s|S]afetynet.*"

You can read the query as "show me all unique user inputs in which the safety net flow was triggered." More literally: in which the name of the flow triggered contains the string "safetynet". This is the default name for the safetynet in Teneo solutions using the Teneo Dialogue Resources. If you're using a safetynet with a different name, be sure to adjust the query accordingly.

By scanning the result, you can find two kinds of inputs:

  1. Questions not yet in the solution's domain
  2. Variants of questions that the solution should answer, but were not assigned to the correct intent.

In the case of 1. you can decide to add new content to the solution. In the case of 2. you can adjust your training data for the intent, or expand the syntax condition to recognize that positive example.

Note: if you want to know inputs to other flows, you can rewrite the query above using the exact name of the particular flow: `lu t.e1.userInput: t.e2.fname == "Name of your flow here"

View flows preceding the safetynet

If a user input hits the safety net, you may want to go one step further and zoom out to get the bigger picture of what happened. A useful query for this is to list the flow and output that preceded the safetynet input. Maybe the bot's answer was unclear, causing the user to ask a follow-up question for clarification. Or perhaps users did not like the answer they got, and reacted with a complaint. In any case, these are solution hotspots where you may want to take action, either by improving the bot's response, or perhaps by adding a followup response in an individual flow.

A query that gives you this information can look like this:

lu t1.id as "Session ID", t1.e1.fname as "Preceding Flow", t1.e2.answerText as "Preceding Bot Response", t2.e1.userInput as "Unrecognized User Input" :
    t1.index == t2.index - 1,
    t1.e1.fname !~ ".*[s|S]afetynet.*",
    t2.e2.fname ~= ".*[s|S]afetynet.*"

The query might appear rather overwhelming at first, so let's walk through what it does:

  1. The command and selection lu t1.id as "Session ID", t1.e1.fname as "Preceding Flow", t1.e2.answerText as "Preceding Bot Response", t2.e1.userInput as "Unrecognized User Input" tells TQL to generate a report with the following inormation: transaction ID, the flow name and response for all transactions that we found. The header column should include the labels as given, e.g. "Preceding Flow", etc.
  2. The constraint t1.index == t2.index - 1 tells TQL, let's only look at pairs of transactions that are immediately consecutive. T1, the first transaction, should happen immediately before T2, the second transaction. We can force this by referencing the indices of the two transactions. The index of T1 should be 1 less than the index of T2.
  3. The constraint t1.e.fname !~ ".[s|S]afetynet." is optional, but you might want it. It means, we want to ignore all transaction pairs in which the safetynet response itself preceded a second response of the safetynet. Possibly these transactions are just noise, but if you want to see them anyway, you can omit the constraint.
  4. The final constraint t2.e2.fname ~= ".[s|S]afetynet." makes sure that the second transaction is one in which the safetynet was triggered.

After running the query your results will look like something like this:

input and flow preceding safetynet

A word about events

Did you notice we refer to items from different events in the query above?

In the first transaction we look at:

  • t1.e1.fname
  • t1.e2.answerText

In the second transaction we look at:

  • t2.e1.userInput
  • t2.e2.fname

There is a subtle and important reason for this: the attributes we query occur at different times (events) during the transaction. In t1: first a flow is raised, then a response is chosen. These belong to different events. If we looked for these elements in the same event, our query would get zero hits, So we must write the query this way, assuring we look at different events in the transaction. Same with the 2nd transaction: the user input is associated with the request event, and the flow that triggers is recorded in one of the later path events. If you're writing a query that you know should get results, and it finds nothing, you may want to review the query to assure that you are referring to the correct events.

Was this page helpful?