Interactive Labeling for ML-based Structural Formula Extraction

Goals

The goal of this Master’s thesis is to design, develop and evaluate an interactive labeling system to support the accessibility of figures. Thereby interactive labeling refers to a human-machine cooperative approach, which combines automatic with manual steps. Structural formulas from the field of chemistry offer themselves as a context of application for this system, as they are frequently used and standards have already been well established. We envision a semi-automated approach, in which user input is supported by the machine. Well structured tasks like these suit themselves well to be supported by machine learning models. As a user is always involved, the model does not need to achieve near-perfect accuracy scores, but rather should support the users with suggestions. Allowing the model to improve with new user input would be a bonus.

In a first step we expect the student to identify the state of the art such systems, and identify components that could be re-used or adapted to this context. Afterwards the solution should be developed. A full-fledged evaluation of the system is expected as well.

The typical workflow for the system should look like the following:

• Import a PDF document into the system.
• The system suggests areas in which figures chemical formulas could be found.
• Correct the systems suggestions.
• Crop out all marked areas to obtain indidual figures.
• For each figure create
  • a chemfig representation of the figure (e.g. “\chemfig{*6(=-=-=-)}”),
  • a non-informative textual description of the figure (e.g. “a hexagon where three edges are double lines”)
  • and an interpretation of the figure (e.g. “Benzene”).
• The system supports the user in creation of above representations with automatically generated suggestions. Hereby a classifier from automatically generated training data that translates images to chemfig should be trained.
• Export an accessible EPUB v3 where the original figure is augmented with above data as alternative versions.
• Export a version of the figure for use on a braille printer (Open Document Graphic format).

Über ISSD - KIT

1825 gegründet

500+ Mitarbeiter

Bildung

Global Player

Karlsruhe

The research group “Information Systems & Service Design” (ISSD) headed by Prof. Mädche focuses in research, education, and innovation on designing interactive intelligent systems. The research belongs to the Institute of Information Systems and Marketing (IISM) and is embedded into the Information Systems & Engineering group. ISSD is also part of the Karlsruhe Service Research Institute (KSRI). The research group is positioned at the intersection of Information Systems (german: Wirtschaftsinformatik) and Human-Computer Interaction (HCI). Our mission is to create impactful scientific knowledge for designing interactive intelligent systems that enable humans to perform activities more efficiently, effectively, and meaningful. We believe that delivering cutting-edge knowledge and inspiring education, as well as an ongoing dialog with the public need to go hand in hand to maximize the impact of our work in organizations and society. The group is organized in three research departments: Digital Experience & Participation, Intelligent Enterprise Systems, and Digital Service Design & Innovation. Current topics of research are Human-AI Interaction, Cognitive Interaction Technologies, Physiological Computing Systems, Interactive Business Intelligence & Analytics Systems, and Interactive Systems Engineering.