LinkedIn trains AI to describe images posted in feed

  • Oct 15, 2019
  • 44

LinkedIn has published on its engineering blog a study by its artificial intelligence department about developing an AI that can automatically generate textual descriptions in user-posted images. The idea is for this technology to be implemented within the platform.

It's been a while since the social network news feed is populated not only by text content, but also by images. And while this feature generally enriches the message the user wants to convey, the format makes it difficult for those with a poor visual quality or poor connection, as this audience can only access the text portion of the post.

You can now enter this manual description when uploading an image, but since most do not choose to include text, the platform is working on a system that can offer alternatives that accurately represent what is pictured in the photo. .

For this mission, LinkedIn used the Analyze API solution, provided by Microsoft Cognitive Services, which is capable of generating multiple descriptions from an image upload.

Creating a solution capable of describing an image is still quite a laborious task within the artificial intelligence universe. This is because, to achieve a quality result, the system requires a massive database and human supervision to pinpoint which models are correct and which ones need to be discarded.

And this task has become even more complicated for LinkedIn, as it deals with objects that are not usually included in the databases that already exist for creating these AIs, such as slides, flipcharts, and big screens.

Because the Analyze API was also trained on a more general basis, the team's first step was to conduct a series of tests to understand how capable the system is to properly process a corporate image.

For this, they created a "confidence score" used by the team, which manually evaluated some of the records presented by the application, and then fine-tune the system.

Next steps
After the initial round of testing, it was already possible to achieve very good results when it comes to describing groups, people performing and internal scenarios.
However, the team is still working to improve the more specific results related to LinkedIn content and to eliminate as much as possible the chance of some content having a very misleading caption. When this step is completed, chances are that the feature will be integrated within the social network.

Related Post