Speech recognition for machine control

Controlling a machine by giving specific spoken commands? Fraunhofer speech recognition software makes it possible. Employees address distant machines directly via their voice instead of their hands in sensitive areas. This possibility is of acute interest in hospitals and operating rooms. Currently, voice control is being tested in production machine environments. We spoke with Marvin Norda and Stefan Wetzel from Fraunhofer IDMT (Germany). Both are involved in a project to make this a reality.

Mr. Norda, in the project "Voice Controlled Production" at Fraunhofer IDMT in Oldenburg, you are developing intuitive voice control for machines that are intended to simplify work in the production process. Can you explain the process?

Marvin Norda: Our speech recognition with all its features can recognize speech in different environments and derive information from it. That's why we don't focus on a specific area of industrial production, because where speech recognition is later used is secondary.

The system creates a text file based on phonemes, which can then be read by any control system, regardless of whether it is a robot, a lathe, an automatic production machine or an automated guided vehicle.

What size is the speech recognition hardware?

Norda: In principle, we focus on the needs of our customers. On the one hand, there is the option of secure online access to our speech recognition technology. In other words, the customer can process the technology on his server or access our Fraunhofer server. On the other hand, the voice control or voice documentation can be processed offline on a minicomputer, for example directly at the machine. So we are very flexible. We have deliberately designed our speech recognition to run on small hardware. It is not necessary to connect a large component. However, it is often the case, especially in the production sector, that very few people want to control their machines via a small computer. Even if it works well and reliably. For example, a Raspberry PI is not a certified control unit for the production area.

That's why many companies use a programmable logic controller (PLC), which is established in the production environment and well-known to testing organizations. However, a rethinking is taking place. Small micro PCs are gradually finding their way into the industrial sector and will therefore sooner or later also be certified for such applications.

Many users of voice controls, for example Google or Alexa, repeatedly experience situations in which what was spoken was not recognized. Sometimes this is due to unclear pronunciation, sometimes because a term is not known. How clearly does the user need to speak with your voice control?

Norda: We have a standard vocabulary that we use and that we train with training material. That is sufficient. Unlike a few years ago, I no longer have to train a speaker either. It's enough that I have the training data to recognize different speakers.

Even if a dialect is spoken, for example?

Norda: It's a question of the desired accuracy. Of course, you can teach the software dialects. But that is usually not necessary. For example, someone who grew up in Bavaria (the far southern state in Germany) will understand a person from Bavaria better than he will understand an East Frisian (a northern state) and an East Frisian will understand an East Frisian better than a Bavarian. But that doesn't mean that a Bavarian and an East Frisian can't understand each other. This is similar with speech recognition. If I provide more training material in this direction, it works better. Just as misunderstandings can happen to an East Frisian in Bavaria, you understand something incorrectly because you have not been trained for it.

Do the users get certain keywords so that they can control the device?

Norda: Yes, I have to know certain commands. We have not integrated a system that would start interpreting the words. It would also be fatal if the machine started interpreting commands on its own. We use keywords that we make available in the customer system and that we work out in dialog with the customers. This is because different companies want different commands. Of course, we offer a standard set of commands. But we also cater to individual wishes. And, of course, we instruct employees on how to operate the system. There are very expensive devices in modern production halls, and of course we don't want the wrong command to be given by mistake.

Do users get feedback when they operate the voice control?

Norda: Yes, on different levels. We've found that interacting with machines doesn't work without some form of feedback. What that feedback looks like depends on the individual machine. For example, if I start a hall-sized machine that starts producing deafening noise by starting it, I automatically have feedback. There's no need for another audio feedback. It's similar when I turn on a light. You can see that immediately.

It's a different matter, for example, to set parameters, then I need audio feedback or visual feedback. We also have hybrid systems in use with customers that have a large screen. On this, for example, changes to parameters are displayed visually.

What hardware do you need to enter the commands?

Norda: A headset, for example. If you have a production hall with computers for the different machines, then you can install the speech recognition on each of them and use it with the headset via a wireless network. Of course, you need a keyword to address the desired machines. Often, however, machines in production already have names, so that a keyword is usually easy to determine.

Does the age of the production machines play a role in the integration of voice control?

Norda: For demonstration purposes, we have rebuilt two machines at the HSA (Hearing, Speech and Audio Technology Department), one of which dates back to 1983. It was built for a diploma thesis at the time. We succeeded in integrating the speech control without any problems. Of course, there are no standard interfaces on machines built before 1980. But it is technically possible. There is no machine age per se that generally rules out voice control. It just has to be electrical. Of course, there is always the question of cost/benefit analysis.

You just spoke briefly about the ambient noise level in a production hall. How can you guarantee that the speech recognition actually understands the spoken words? How do you make it so that the device only filters out what is directed at the machine?

Norda: That is indeed an important point. Robustness, i.e. reliability, and efficiency are the two big issues for the industrialization of voice control. As I said, our voice control system receives training, that is, a certain set of words. This means that our speech recognition system, which is used by the customer, no longer knows all the words of the German language, but is trimmed to exactly what is needed for the individual company. As a result, it is given blinders, so to speak. This is particularly important for other applications, such as hospitals. Speech recognition can also make sense in this environment. But of course, speech recognition must then only respond to me as the user and not to the colleague next to me who may be talking to a patient. This means that speech recognition is focused on reacting only when the right word is spoken, or the right commands are given. This makes our speech recognition very different from applications in the consumer sector, such as Alexa. Here, a "normal" conversation is more desirable. But if I agree from the start to use only sentences that can be clearly distinguished from each other, then I reduce misunderstandings.

In addition to production equipment, some exhibitors at COMPAMED also offer end products that can be used in hospitals. Can voice control also be used profitably here?

Stefan Wetzel: Basically yes, because we are talking about voice control of machines or devices, regardless of their area of application. In principle, similar requirements apply everywhere. Voice control is also conceivable for medical devices.

Do you have an example of an area of application?

Wetzel: The fields of application are almost unlimited. Of course, you always must look at the individual case and ask yourself: Where does voice control make sense? In some cases, it is certainly easier and better to carry out human-machine interaction in the conventional way. But wherever voice control offers an additional benefit, it could be used. I'm thinking, for example, of hygienically sensitive areas, such as the operating room. Instead of using hands, for example, lighting or controlling certain devices could be done by voice control. Or the documentation of the operation could be done simultaneously, with the doctor speaking all the essentials. Then there would be no need to sit down at a PC later and type everything in. We have already carried out projects with partners to simplify documentation in nursing, for example. The requirements in the care sector are different from those in production or in hospitals. In general, we always must consider the field in which we are operating. But you can see from this that speech recognition can be used in various areas.

Do you see any potential problems with regard to valid regulations when it comes to medical technology?

Wetzel: Regulation for the hospital sector could be more complex than for industrial production - but it doesn't have to be in individual cases, which certainly depends on the application scenario. In addition, it would have to be clarified whether voice control is already considered a medical device. Probably not. For this purpose, and to define the use cases even more clearly, we are currently seeking dialog with manufacturers and end users.