<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom"><generator uri="https://jekyllrb.com/" version="4.3.3">Jekyll</generator><link href="https://defret.in/feed.xml" rel="self" type="application/atom+xml"/><link href="https://defret.in/" rel="alternate" type="text/html"/><updated>2024-09-12T07:51:11+00:00</updated><id>https://defret.in/feed.xml</id><title type="html">Yann Defretin</title><subtitle>Some articles posted by Yann Defretin, Developer and AI enthusiast.</subtitle><author><name>Yann Defretin</name></author><entry><title type="html">Add a Dark Theme to a Ruby on Rails app</title><link href="https://defret.in/log/dark-theme-on-ruby-on-rails" rel="alternate" type="text/html" title="Add a Dark Theme to a Ruby on Rails app"/><published>2021-03-18T00:00:00+00:00</published><updated>2021-03-18T00:00:00+00:00</updated><id>https://defret.in/log/add-dark-theme-on-ruby-on-rails-app</id><content type="html" xml:base="https://defret.in/log/dark-theme-on-ruby-on-rails"><![CDATA[<p>Dark themes recently attracted a lot of attention since enterprises such as Apple or Google deployed on most of their operating systems — iOS, macOS, Android —, an alternative to the default interface colors which were mainly light.</p> <p>Microsoft followed the trend and added the same option on <a href="https://www.pcmag.com/how-to/how-to-enable-dark-mode-in-windows-10">Windows</a>. While it is a great feature to allow people to use their computer or mobile at night without hurting their eyes, <strong>this implementation is incomplete.</strong> Indeed, to make this feature really shine, having the operating system interface dark is not enough. Developers must follow the rules too and add a dark theme to their softwares. Otherwise, it’s pointless.</p> <p>More importantly, web developers have also to think about adding a Dark Theme option to their websites otherwise, no matter if your OS interface is light or dark, the website will still look as before. And if, just like the homepage of Google, it is full of white and very bright, it would be awful at night to browse, even more if your OS interface is set to dark.</p> <p>Fortunately, it is easy to quickly create a dark version of a website, thanks to the “<a href="https://developer.mozilla.org/en-US/docs/Web/CSS/@media/prefers-color-scheme">prefers-color-scheme</a>” property added to the CSS specifications.</p> <h2 id="detection-of-the-operating-system-colors">Detection of the Operating System colors</h2> <p>The CSS media feature <code class="language-plaintext highlighter-rouge">prefers-color-scheme</code> can take two values: either “light” or “dark”. It allows web developers to adjust the design of their website if the Operating System interface of the user is set to the default one — most of the time, the “light” one —, or to the “dark” one.</p> <p>For example, if we want to change the background of our website to black if the user has the Dark Mode option enabled on its OS, we can simply do:</p> <div class="language-css highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">/* style.css */</span>
<span class="c">/* change background color if the OS is on Dark Mode */</span>
<span class="k">@media</span> <span class="p">(</span><span class="n">prefers-color-scheme</span><span class="p">:</span> <span class="n">dark</span><span class="p">)</span> <span class="p">{</span>
  <span class="nt">body</span> <span class="p">{</span>
    <span class="nl">background-color</span><span class="p">:</span> <span class="m">#000</span> <span class="cp">!important</span><span class="p">;</span>
  <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div> <p>This is very handy as we can now detect the interface mode of the user and adapt the colors of our website to match it.</p> <p>More interestingly, this CSS feature will detect the OS interface change instantly. For example, on macOS, you can schedule your OS to automatically switch the interface from light to dark at sunset. And the contrary at sunrise.</p> <p>Every app will then change its colors too. This is very cool. And if this switch happens when you are browsing a website that uses <code class="language-plaintext highlighter-rouge">prefers-color-scheme</code>, it will use the dark colors automatically, without having to refresh the page.</p> <p>Now, how to implement that nicely on an existing Ruby on Rails app?</p> <h2 id="dark-mode-on-rails-with-scss">Dark Mode on Rails with SCSS</h2> <p>For most of my web projects, I like to use Ruby on Rails either for quick prototyping, either for real projects deployed in production. This is the case for <a href="https://whatthemovie.com/shot/random">whatthemovie.com</a>, a movie quiz game, where people upload movie snapshots and the other players have to guess from what movie the image was taken.</p> <p><a href="/assets/images/2021-03-18-add-dark-theme-on-ruby-on-rails-app/1*TtLDnMcl3_tTcfd5lM40og.png"><picture><source sizes="(max-width: 480px) 80vw, (max-width: 979px) 81vw, (max-width: 1199px) 82vw, (min-width: 1200px) 83vw" srcset="/generated/assets/images/2021-03-18-add-dark-theme-on-ruby-on-rails-app/1*TtLDnMcl3_tTcfd5lM40og-200-e94a8256e.webp 200w, /generated/assets/images/2021-03-18-add-dark-theme-on-ruby-on-rails-app/1*TtLDnMcl3_tTcfd5lM40og-400-e94a8256e.webp 400w, /generated/assets/images/2021-03-18-add-dark-theme-on-ruby-on-rails-app/1*TtLDnMcl3_tTcfd5lM40og-600-e94a8256e.webp 600w, /generated/assets/images/2021-03-18-add-dark-theme-on-ruby-on-rails-app/1*TtLDnMcl3_tTcfd5lM40og-700-e94a8256e.webp 700w" type="image/webp"><source sizes="(max-width: 480px) 80vw, (max-width: 979px) 81vw, (max-width: 1199px) 82vw, (min-width: 1200px) 83vw" srcset="/generated/assets/images/2021-03-18-add-dark-theme-on-ruby-on-rails-app/1*TtLDnMcl3_tTcfd5lM40og-200-e94a8256e.png 200w, /generated/assets/images/2021-03-18-add-dark-theme-on-ruby-on-rails-app/1*TtLDnMcl3_tTcfd5lM40og-400-e94a8256e.png 400w, /generated/assets/images/2021-03-18-add-dark-theme-on-ruby-on-rails-app/1*TtLDnMcl3_tTcfd5lM40og-600-e94a8256e.png 600w, /generated/assets/images/2021-03-18-add-dark-theme-on-ruby-on-rails-app/1*TtLDnMcl3_tTcfd5lM40og-700-e94a8256e.png 700w" type="image/png"><img src="/generated/assets/images/2021-03-18-add-dark-theme-on-ruby-on-rails-app/1*TtLDnMcl3_tTcfd5lM40og-700-e94a8256e.png" alt="Guess the movie behind the shot" width="700" height="570"></picture></a> <em>Can you guess the movie behind this snapshot?</em></p> <div class="premonition info"> <i class="premonition pn-info"></i> <div class="content"> <p>If you are interested, I made <a href="/log/create-dataset-multilabel-model">two</a> <a href="/log/train-multilabel-image-model-movies"> articles</a> about this project where I explain how I created a dataset and an AI model to suggest tags on what the movie snapshots contain or are.</p> </div> </div> <p>Like many other Rails apps, it heavily uses the <a href="https://guides.rubyonrails.org/asset_pipeline.html">Assets Pipeline</a> to handle assets — images, javascript, fonts, stylesheets etc.</p> <p>Unfortunately, our design is quite outdated and not modern at all — it is from 2008, literally. For example, our buttons were made out of images to have colors gradients where nowadays, you can simply use the <code class="language-plaintext highlighter-rouge">linear-gradient</code> CSS function to do the same, in one line! My point being: it will be more difficult and need more work if your design is old.</p> <p>If you use for example CSS variables, you will see that it will be more easier to create a dark mode and quicker because you basically just have to change the colors of these variables and you are done.</p> <h2 id="create-a-dedicated-dark-mode-scss-file">Create a dedicated Dark Mode SCSS file</h2> <p>The first thing we have to do is to keep files organized and respect the Assets Pipeline guide. Therefore, to create our Dark Theme, we will add a new file named <code class="language-plaintext highlighter-rouge">darkmode.scss</code> to the Pipeline in <code class="language-plaintext highlighter-rouge">app/assets/stylesheets</code>.</p> <p>We also have to make sure our Rails app will now integrate this file and its content to the CSS file we will have in production. To do that, we add a reference to our new SCSS file in <code class="language-plaintext highlighter-rouge">application.scss</code>.</p> <figure class="highlight"><pre><code class="language-scss" data-lang="scss"><span class="cm">/* app/assets/stylesheets/application.scss */</span>

<span class="cm">/*</span>
<span class="cm">*= require main</span>
<span class="cm">[...]</span>
<span class="hll"><span class="cm">*= require darkmode</span>
</span><span class="cm">*/</span></code></pre></figure> <p>Make sure to require the SCSS file at the end, after everything else is already included. Since we are going to change or overwrite styles, it is important to add the Dark Theme after everything is set up.</p> <p>Now it is time to stylize our website with some dark colors! For that, we are going to use a combination of SCSS functions such as <code class="language-plaintext highlighter-rouge">mixin</code> and <code class="language-plaintext highlighter-rouge">include</code>, HTML data attribute and a ruby gem to save and retrieve the user preference.</p> <p>See the SCSS <code class="language-plaintext highlighter-rouge">mixin</code> feature as a way to define styles that you wish to re-use later one or more times without having to maintain numerous versions of the same thing. We are going to create a mixin called “dark” that will contain any change that makes our elements, selectors, identifiers, darker.</p> <div class="language-scss highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/* app/assets/stylesheets/darkmode.scss */</span>

<span class="cm">/* change background color if the OS is on Dark Mode */</span> 
<span class="k">@mixin</span> <span class="nf">dark</span> <span class="p">{</span>
  <span class="nt">body</span> <span class="p">{</span>
    <span class="nl">background-color</span><span class="p">:</span> <span class="mh">#000</span> <span class="o">!</span><span class="n">important</span><span class="p">;</span>
  <span class="p">}</span>
<span class="p">}</span>

<span class="k">@media</span> <span class="p">(</span><span class="n">prefers-color-scheme</span><span class="o">:</span> <span class="n">dark</span><span class="p">)</span> <span class="p">{</span>
  <span class="k">@include</span> <span class="nd">dark</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div> <p>The content of <code class="language-plaintext highlighter-rouge">mixin</code> “dark” will automatically be included in the section where it is called. This is a silly example to introduce the concept.</p> <p>Now, we need to think about how handle what the user really wants. Every possible scenario.</p> <h2 id="three-options-automatic-light-dark">Three options: automatic, light, dark</h2> <p>While creating a dark version of our website that will follow the user OS interface colors sounds nice, it might not respect what the user really want.</p> <p>For example, a user might want to use its OS interface with light colors but browse the website with the Dark Theme enabled. Same in the opposite direction: a user with its OS interface set to dark might still want to browse our website with the old light theme.</p> <p>Therefore, we have three scenarios to handle and to offer as options to the users:</p> <ul> <li> <p><strong>Automatic Mode:</strong> This option describes what we talked about earlier. The website will follow the user OS interface colors automatically. If the OS interface is set to light, the theme on the website is light. If it’s dark, the theme will be dark ;</p> </li> <li> <p><strong>Light Theme:</strong> This option forces the use of the old light theme, no matter what the user OS interface is set to ;</p> </li> <li> <p><strong>Dark Theme:</strong> This option forces the use of the new dark theme, no matter what the user OS interface is set to. This is handy as many users don’t run on an OS that provides a light and dark mode. Therefore, if they want to use the dark theme, it is the only way.</p> </li> </ul> <p>This is where SCSS features and HTML data attributes really shine. To handle these three scenarios, we will use a data attribute called <code class="language-plaintext highlighter-rouge">data-color-mode</code> that can take three values : “auto”, “light” or “dark” — these values will reflect the user preference. And we are going to use it, in combination with <code class="language-plaintext highlighter-rouge">mixin</code>, <code class="language-plaintext highlighter-rouge">include</code> and <code class="language-plaintext highlighter-rouge">prefers-color-scheme</code>, to offer this dark theme to everybody:</p> <div class="language-scss highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/* app/assets/stylesheets/darkmode.scss */</span>

<span class="k">@mixin</span> <span class="nf">dark</span> <span class="p">{</span>
<span class="cm">/* all the colors/css changes to make the website dark */</span>
<span class="p">}</span>

<span class="cm">/* force the dark mode when OS interface is light */</span>
<span class="k">@media</span> <span class="p">(</span><span class="n">prefers-color-scheme</span><span class="o">:</span> <span class="n">light</span><span class="p">)</span> <span class="p">{</span>
  <span class="o">[</span><span class="nt">data-color-mode</span><span class="o">=</span><span class="s2">"dark"</span><span class="o">]</span> <span class="p">{</span>
    <span class="k">@include</span> <span class="nd">dark</span><span class="p">;</span>
  <span class="p">}</span>
<span class="p">}</span>

<span class="cm">/* follow the OS interface colors or force the dark theme */</span>
<span class="k">@media</span> <span class="p">(</span><span class="n">prefers-color-scheme</span><span class="o">:</span> <span class="n">dark</span><span class="p">)</span> <span class="p">{</span>
  <span class="o">[</span><span class="nt">data-color-mode</span><span class="o">=</span><span class="s2">"auto"</span><span class="o">],</span> <span class="o">[</span><span class="nt">data-color-mode</span><span class="o">=</span><span class="s2">"dark"</span><span class="o">]</span> <span class="p">{</span>
    <span class="k">@include</span> <span class="nd">dark</span><span class="p">;</span>
  <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div> <p>With this code snippet, we now handle the three possible scenarios: if the user has an OS interface set to light but want to use the dark theme anyway etc. We now have to let the user pick what option it prefers, saves it, retrieves it and sets it inside the HTML data attribute <code class="language-plaintext highlighter-rouge">data-color-mode</code>.</p> <h2 id="store-and-retrieve-user-preferences-regarding-appearance">Store and retrieve user preferences regarding appearance</h2> <p>To keep the user settings on our website, we use the Ruby Gem <a href="https://github.com/pluginaweek/preferences">preferences</a>. It is very easy and simple to use. I highly recommend it. To add a user setting to store, we add this line to the user model:</p> <div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># app/models/user.rb</span>

<span class="p">[</span><span class="o">...</span><span class="p">]</span>
<span class="n">preference</span> <span class="ss">:theme</span><span class="p">,</span> <span class="ss">:string</span><span class="p">,</span> <span class="ss">default: </span><span class="s1">'auto'</span>
</code></pre></div></div> <p>With this line, we store the user preference about the website design as “theme” and set it to “auto” by default.</p> <p>Now let’s edit the user settings form to add the ability to change the theme and save the user choice:</p> <div class="language-erb highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">&lt;!-- app/views/user/settings.html.erb --&gt;</span>

[...]

<span class="nt">&lt;h2&gt;</span>Appearance<span class="nt">&lt;/h2&gt;</span>

<span class="nt">&lt;p&gt;</span>Choose how WTM looks to you. Set your theme preference to follow your system settings, or choose to use always the light or dark theme.<span class="nt">&lt;/p&gt;</span>

<span class="nt">&lt;ul</span> <span class="na">class=</span><span class="s">"standard_form"</span><span class="nt">&gt;</span>
  <span class="nt">&lt;li&gt;</span>
    <span class="nt">&lt;div</span> <span class="na">class=</span><span class="s">"title"</span><span class="nt">&gt;</span>Theme<span class="nt">&lt;/div&gt;</span>
    <span class="nt">&lt;div</span> <span class="na">class=</span><span class="s">"content"</span><span class="nt">&gt;</span>
      <span class="cp">&lt;%=</span> <span class="n">f</span><span class="p">.</span><span class="nf">radio_button</span><span class="p">(</span><span class="ss">:prefers_theme</span><span class="p">,</span> <span class="s2">"auto"</span><span class="p">)</span> <span class="cp">%&gt;</span> 
      <span class="nt">&lt;label</span> <span class="na">for=</span><span class="s">"user_theme_auto"</span><span class="nt">&gt;</span>Default to System<span class="nt">&lt;/label&gt;</span>
      <span class="cp">&lt;%=</span> <span class="n">f</span><span class="p">.</span><span class="nf">radio_button</span><span class="p">(</span><span class="ss">:prefers_theme</span><span class="p">,</span> <span class="s2">"light"</span><span class="p">)</span> <span class="cp">%&gt;</span> 
      <span class="nt">&lt;label</span> <span class="na">for=</span><span class="s">"user_theme_light"</span><span class="nt">&gt;</span>Light<span class="nt">&lt;/label&gt;</span>
      <span class="cp">&lt;%=</span> <span class="n">f</span><span class="p">.</span><span class="nf">radio_button</span><span class="p">(</span><span class="ss">:prefers_theme</span><span class="p">,</span> <span class="s2">"dark"</span><span class="p">)</span> <span class="cp">%&gt;</span> 
      <span class="nt">&lt;label</span> <span class="na">for=</span><span class="s">"user_theme_dark"</span><span class="nt">&gt;</span>Dark<span class="nt">&lt;/label&gt;</span>
    <span class="nt">&lt;/div&gt;</span>
  <span class="nt">&lt;/li&gt;</span>
<span class="nt">&lt;/ul&gt;</span>
</code></pre></div></div> <p>It looks like this:</p> <p><a href="/assets/images/2021-03-18-add-dark-theme-on-ruby-on-rails-app/1*TiSBegFhOT6KtPn-dsHoMw.png"><picture><source sizes="(max-width: 480px) 80vw, (max-width: 979px) 81vw, (max-width: 1199px) 82vw, (min-width: 1200px) 83vw" srcset="/generated/assets/images/2021-03-18-add-dark-theme-on-ruby-on-rails-app/1*TiSBegFhOT6KtPn-dsHoMw-200-1f4957446.webp 200w, /generated/assets/images/2021-03-18-add-dark-theme-on-ruby-on-rails-app/1*TiSBegFhOT6KtPn-dsHoMw-400-1f4957446.webp 400w, /generated/assets/images/2021-03-18-add-dark-theme-on-ruby-on-rails-app/1*TiSBegFhOT6KtPn-dsHoMw-593-1f4957446.webp 593w" type="image/webp"><source sizes="(max-width: 480px) 80vw, (max-width: 979px) 81vw, (max-width: 1199px) 82vw, (min-width: 1200px) 83vw" srcset="/generated/assets/images/2021-03-18-add-dark-theme-on-ruby-on-rails-app/1*TiSBegFhOT6KtPn-dsHoMw-200-1f4957446.png 200w, /generated/assets/images/2021-03-18-add-dark-theme-on-ruby-on-rails-app/1*TiSBegFhOT6KtPn-dsHoMw-400-1f4957446.png 400w, /generated/assets/images/2021-03-18-add-dark-theme-on-ruby-on-rails-app/1*TiSBegFhOT6KtPn-dsHoMw-593-1f4957446.png 593w" type="image/png"><img src="/generated/assets/images/2021-03-18-add-dark-theme-on-ruby-on-rails-app/1*TiSBegFhOT6KtPn-dsHoMw-593-1f4957446.png" alt="The user settings page" width="593" height="129"></picture></a> <em>The user settings page</em></p> <p>The last thing to do is to add our HTML data attribute <code class="language-plaintext highlighter-rouge">data-color-mode</code> in our main application layout and set its value to the user preference we retrieved from the database in order to make our SCSS features work:</p> <div class="language-erb highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">&lt;!-- app/views/layouts/application.html.erb --&gt;</span>

<span class="nt">&lt;html</span> <span class="na">data-color-mode=</span><span class="s">"</span><span class="cp">&lt;%=</span> <span class="p">(</span><span class="vi">@current_user</span><span class="o">&amp;</span><span class="p">.</span><span class="nf">preferred_theme</span> <span class="o">||</span> <span class="s2">"auto"</span><span class="p">)</span> <span class="cp">%&gt;</span><span class="s">"</span><span class="nt">&gt;</span>
</code></pre></div></div> <p>We make use of the <a href="https://mitrev.net/ruby/2015/11/13/the-operator-in-ruby/">Safe Navigation Operator</a> <code class="language-plaintext highlighter-rouge">&amp;.</code> introduced in Ruby 2.3 to handle the case where the current visitor/user is not logged in to our website. If it is not logged in, it will return <code class="language-plaintext highlighter-rouge">nil</code> and set the <code class="language-plaintext highlighter-rouge">data-color-mode</code> value to “auto”. If the current user is logged in, it will retrieve its preference regarding the theme of the website and set the correct value in the data attribute.</p> <p>And now we are done! We have added a Dark Theme to our website!</p> <p><img src="/assets/images/2021-03-18-add-dark-theme-on-ruby-on-rails-app/ezgif-4-36cfbd5856c8.gif" alt="Our new dark theme"/> <em>Our new dark theme!</em></p> <h2 id="some-caveats">Some caveats</h2> <p>Since our design is very old, this approach was quite nice and easy to implement. Unfortunately, it comes with some weird behaviors. Indeed, the CSS file generated by SCSS and the Assets Pipeline is, in our case, very large.</p> <p>It seems the compiler needs to add in front of every element, selector or identifier, the data attribute, like this:</p> <div class="language-css highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">/* generated CSS file */</span>

<span class="o">[</span><span class="nt">data-color-mode</span><span class="o">=</span><span class="s1">"auto"</span><span class="o">]</span> <span class="nt">ul</span><span class="nc">.disclist</span> <span class="nt">li</span> <span class="nt">a</span><span class="o">,</span>
<span class="o">[</span><span class="nt">data-color-mode</span><span class="o">=</span><span class="s1">"auto"</span><span class="o">]</span> <span class="nc">.profile_info</span> <span class="nt">a</span><span class="o">,</span>
<span class="o">[</span><span class="nt">data-color-mode</span><span class="o">=</span><span class="s1">"auto"</span><span class="o">]</span> <span class="nc">.profile_user_links</span> <span class="nt">li</span> <span class="nt">a</span><span class="o">,</span>
<span class="o">[</span><span class="nt">data-color-mode</span><span class="o">=</span><span class="s1">"auto"</span><span class="o">]</span> <span class="nc">.thumb_overlay</span> <span class="nt">a</span> <span class="nt">span</span><span class="o">,</span>
<span class="o">[</span><span class="nt">data-color-mode</span><span class="o">=</span><span class="s1">"auto"</span><span class="o">]</span> <span class="nt">li</span><span class="nc">.sotd</span> <span class="nt">h4</span> <span class="nt">a</span><span class="o">,</span>
<span class="o">[</span><span class="nt">data-color-mode</span><span class="o">=</span><span class="s1">"auto"</span><span class="o">]</span> <span class="nf">#online_users</span> <span class="nc">.footer_box</span> <span class="nt">a</span><span class="o">,</span>

<span class="o">[</span><span class="nt">data-color-mode</span><span class="o">=</span><span class="s1">"dark"</span><span class="o">]</span> <span class="nt">ul</span><span class="nc">.disclist</span> <span class="nt">li</span> <span class="nt">a</span><span class="o">,</span>
<span class="o">[</span><span class="nt">data-color-mode</span><span class="o">=</span><span class="s1">"dark"</span><span class="o">]</span> <span class="nc">.profile_info</span> <span class="nt">a</span><span class="o">,</span>
<span class="o">[</span><span class="nt">data-color-mode</span><span class="o">=</span><span class="s1">"dark"</span><span class="o">]</span> <span class="nc">.profile_user_links</span> <span class="nt">li</span> <span class="nt">a</span><span class="o">,</span>
<span class="o">[</span><span class="nt">data-color-mode</span><span class="o">=</span><span class="s1">"dark"</span><span class="o">]</span> <span class="nc">.thumb_overlay</span> <span class="nt">a</span> <span class="nt">span</span><span class="o">,</span>
<span class="o">[</span><span class="nt">data-color-mode</span><span class="o">=</span><span class="s1">"dark"</span><span class="o">]</span> <span class="nt">li</span><span class="nc">.sotd</span> <span class="nt">h4</span> <span class="nt">a</span><span class="o">,</span>
<span class="o">[</span><span class="nt">data-color-mode</span><span class="o">=</span><span class="s1">"dark"</span><span class="o">]</span> <span class="nf">#online_users</span> <span class="nc">.footer_box</span> <span class="nt">a</span> <span class="p">{</span>
    <span class="nl">color</span><span class="p">:</span> <span class="m">#7C7C7C</span> <span class="cp">!important</span><span class="p">;</span>
  <span class="p">}</span>
</code></pre></div></div> <p>It results to a file with a lot of lines generated. While this might not be that much a problem to us, it might be to you. Unless, as we talked about it earlier, your design is well done and makes use of CSS variables. In that case, the generated code will be very small since you just have to changes the CSS variables values.</p> <p>Even if I don’t recommend it, I found a way to optimize the compiled CSS file by <em>combining</em> the data attributes when the value is set to “auto” or “dark”.</p> <p>Instead of checking for the values “auto” and “dark”, I only check for the presence of the letter “a” with <code class="language-plaintext highlighter-rouge">[data-color-mode*="a"]</code>:</p> <figure class="highlight"><pre><code class="language-scss" data-lang="scss"><span class="cm">/* app/assets/stylesheets/darkmode.scss */</span>

<span class="k">@mixin</span> <span class="nf">dark</span> <span class="p">{}</span>

<span class="k">@media</span> <span class="p">(</span><span class="n">prefers-color-scheme</span><span class="o">:</span> <span class="n">light</span><span class="p">)</span> <span class="p">{</span>
<span class="hll">  <span class="o">[</span><span class="nt">data-color-mode</span><span class="o">*=</span><span class="s2">"d"</span><span class="o">]</span> <span class="p">{</span>
</span>    <span class="k">@include</span> <span class="nd">dark</span><span class="p">;</span>
  <span class="p">}</span>
<span class="p">}</span>

<span class="k">@media</span> <span class="p">(</span><span class="n">prefers-color-scheme</span><span class="o">:</span> <span class="n">dark</span><span class="p">)</span> <span class="p">{</span>
<span class="hll">  <span class="o">[</span><span class="nt">data-color-mode</span><span class="o">*=</span><span class="s2">"a"</span><span class="o">]</span> <span class="p">{</span>
</span>    <span class="k">@include</span> <span class="nd">dark</span><span class="p">;</span>
  <span class="p">}</span>
<span class="p">}</span></code></pre></figure> <p>This trick makes use of the <code class="language-plaintext highlighter-rouge">*=</code> attribute selector and basically says: if the <code class="language-plaintext highlighter-rouge">data-color-mode</code> value contains the letter “a”, then do something. Since both “auto” and “dark” contains this letter but not “light”, it works!</p> <p>I did the same for <code class="language-plaintext highlighter-rouge">[data-color-mode="dark"]</code> alone to reduce the number of letters by a few. This little method lead to the removal of more than 400 lines.</p> <p>I posted on <a href="https://stackoverflow.com/questions/66302163/optimize-scss-code-for-dark-theme-integration">StackOverflow</a> asking for help to optimize this without having to remake all my design but so far, no answers. If you have any ideas, please share them!</p> <h2 id="accessibility">Accessibility</h2> <p>Something you have to keep in mind when you start making a dark theme for your website is accessibility. There is nothing more painful to read a text on a dark background when the text color was badly chosen.</p> <p>When you create your Dark Mode Color Palette, think about it: does the text will be readable? Does the contrast between these two elements is enough?</p> <p>GitHub, who recently released a Dark Theme for their website, received <a href="https://blog.karenying.com/posts/github-darkmode-sucks">some criticisms</a> about this because of the lack of contrast between some elements.</p> <p>Fortunately, the <a href="https://www.w3.org/WAI/standards-guidelines/wcag/">Web Content Accessibility Guidelines</a> (WCAG) helps us to choose the correct colors by calculating the contrast ratio between elements.</p> <p>They have defined two levels of accessibility regarding this:</p> <ul> <li> <p>The level <strong>AA:</strong> it needs a contrast ratio as above <strong>4.5</strong> ;</p> </li> <li> <p>The level <strong>AAA:</strong> as above <strong>7</strong> for small text.</p> </li> </ul> <p>Where the level <strong>AAA</strong> should be the objective, acheving the level <strong>AA</strong> is already nice<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>.</p> <p>To help us calculate the contrast ratio between elements, you can use the Developer Tools of your browser: when you open the color picker of a text element, you can see the contrast ratio with its background and the two levels thresholds.</p> <p>If your browser doesn’t have such a feature, multiple websites like <a href="https://webaim.org/resources/contrastchecker/">WebAIM</a> or <a href="https://coolors.co/contrast-checker/112a46-acc8e5">Coolors</a> provide a way to calculate easily the contrast ratio between two colors.</p> <h2 id="conclusion">Conclusion</h2> <p>Without that much work, we were able to add a Dark Theme to our existing Ruby on Rails application. Despite not being a web developer or a web designer, our little theme quickly made seems to have gained some success: more than 74% of our active users have enabled it by default, and it doesn’t include users who enabled the “automatic” mode that automatically switches the theme from “light” to “dark” and vice versa.</p> <p>When you think about it, it makes sense that a website about movies, displaying movie snapshots with cinema benches and curtains around them, provides a better experience to the users with dark colors everywhere.</p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1" role="doc-endnote"> <p>Of course if you can respect both levels, this is awesome. But I found out, in my case, that trying to achieve the level AAA for small texts was sometimes very hard. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div>]]></content><author><name>Yann Defretin</name></author><category term="Rails"/><summary type="html"><![CDATA[How quickly add a Dark Theme alternative to an existing Ruby on Rails application using SCSS and the Assets Pipeline.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://defret.in/assets/images/2021-03-18-add-dark-theme-on-ruby-on-rails-app/1*TtLDnMcl3_tTcfd5lM40og.png"/><media:content medium="image" url="https://defret.in/assets/images/2021-03-18-add-dark-theme-on-ruby-on-rails-app/1*TtLDnMcl3_tTcfd5lM40og.png" xmlns:media="http://search.yahoo.com/mrss/"/></entry><entry><title type="html">Train a multi-label image classification for movie snapshots</title><link href="https://defret.in/log/train-multilabel-image-model-movies" rel="alternate" type="text/html" title="Train a multi-label image classification for movie snapshots"/><published>2019-07-09T00:00:00+00:00</published><updated>2019-07-09T00:00:00+00:00</updated><id>https://defret.in/log/train-multi-label-image-classification</id><content type="html" xml:base="https://defret.in/log/train-multilabel-image-model-movies"><![CDATA[<p>Through our <a href="/log/create-dataset-multilabel-model">previous article</a>, we successfully created a dataset that combines movie snapshots and their respective tags describing what the image contains.</p> <p>As a reminder, the dataset consists of 243 558 images with 183 different labels. It is a simple CSV file that looks like this:</p> <table> <thead> <tr> <th>id</th> <th>shot_image_id</th> <th>tags</th> </tr> </thead> <tbody> <tr> <td>1</td> <td>1</td> <td>statue</td> </tr> <tr> <td>2</td> <td>81091</td> <td>bed, legs, text</td> </tr> <tr> <td>7</td> <td>250992</td> <td>hand, text</td> </tr> <tr> <td>8</td> <td>81360</td> <td>castle, sky</td> </tr> <tr> <td>10</td> <td>9</td> <td>water, redhead</td> </tr> </tbody> </table> <p><em>Our final dataset: an image ID and its tags.</em></p> <p>We are now going to use this dataset to create and train an AI that will be able to detect and suggest elements of the images uploaded by users. But how to do that? What library can we pick to make this AI? Where are we going to find the CPU and GPU ressources needed to train our AI quickly?</p> <p>In this second part of the series, we are first going to find and create an environment suitable for this heavy task. Then, we will pick <a href="https://www.fast.ai">FastAI</a>, one of the most promising deep learning librairies available nowadays, to make our AI and hopefully have, at the end of the article, a good model that we could export in order to use it on our website.</p> <h2 id="set-up-the-environment">Set-up the environment</h2> <p>To train our AI with FastAI, we need to find the best environment available that can handle working over 250 000 images in a short time. Our website runs on two dedicated servers with great hardware to serve a Ruby on Rails app. But Computer Vision needs a lot of ressources, especially a good GPU — which our servers don’t have. Therefore, we need to find a platform that can give us these ressources for a cheap price or even for free, <em>on demand</em>.</p> <p>That’s fortunate since there are two main platforms online which can provide in a very easy way the environment we are looking for.</p> <h3 id="kaggle-a-place-for-data-science-projects">Kaggle, a place for data science projects</h3> <p><a href="https://kaggle.com">Kaggle</a> is a well-known community website for data scientists to compete in deep learning challenges. After joining a competition, you are given a dataset and you have to create an AI that gives the best results on it. You are free to use any deep learning library you wish to.</p> <p>But say that Kaggle is only about competitions would be incorrect. In fact, Kaggle also provides to data scientists hundreds of datasets for free to work on, courses to learn the most used tools in the field and more importantly, a ready-to-use environment with great ressources <strong>for free</strong>.</p> <p>You can then join competitions and fight with other data scientists directly on their website. They give you everything you need: a Jupyter-like interface, some disk space, a good CPU and a GPU. Within a few clicks, you can start training an AI!</p> <p>FastAI, the library we are about to use for this project, is available on Kaggle as well. Actually, this project was heavily inspired by <a href="https://www.kaggle.com/hortonhearsafoo/fast-ai-v3-lesson-3-planet">this kernel</a> that you can fork where Jeremy Howard — the creator of FastAI — , use his library to participate to the famous <a href="https://www.kaggle.com/c/planet-understanding-the-amazon-from-space/overview/description">Planet: Understanding the Amazon from Space</a> competition. He got excellent results with only a few lines of code.</p> <p>You can find a <a href="https://course.fast.ai/start_kaggle.html">nice tutorial</a> on how to start using FastAI on Kaggle on their official website.</p> <p>While Kaggle seems to be the perfect place for our project, it has some limitations. I see Kaggle as a nice website where you can play and try different kind of things. But for serious deep learning projects, you will have to switch to a real provider that gives you more flexibility, more ressources.</p> <h3 id="google-cloud-platform">Google Cloud Platform</h3> <p>Since we want to be able to export our model and re-train our AI every month in an automatic way, we need to find a better host for our task where we can deeply custom our environment according to our needs. <a href="https://cloud.google.com">Google Cloud Platform</a> seems to be the perfect place for that matter. It provides many services for a reasonable price. You can even get 300$ of credits for free for one year when registering.</p> <p>Working with GCP definitely requires more time and more work. You basically have to create a Virtual Machine and install everything you need for your project. The main advantage though is that you are free to create any kind of Virtual Machines for your project. You can set-up a VM with large ressources and even pick the region across the world where the VM would be located.</p> <p>Fortunately, there are plenty of tutorials available online to get you started on GCP. FastAI has a <a href="https://course.fast.ai/start_gcp.html">dedicated tutorial</a> on how to start using their library on the Google Cloud Platform. After following their instructions, you should have an environment with all the tools necessary to start training an AI.</p> <p>Having our environment on the cloud will definitely make things easier when we would have to re-train our model on monthly basis. GCP provides APIs for almost every service they offer including creating, starting and stopping VMs. That’s a really important feature that will make the re-training less difficult.</p> <h2 id="create-an-ai-with-fastai">Create an AI with FastAI</h2> <p>We created a VM running on Google Cloud Platform with a great GPU — a NVIDIA Tesla P100 with 16 gigabytes of VRAM in our case. Just like in our previous article where we created our dataset with Pandas, we are going to use Jupyter to interact with FastAI. But first, let’s talk about this awesome library.</p> <p>The FastAI library offers a high-level API capable of creating deep learning models for a lot of different applications including text generation, text analysis, image classification and image segmentation. FastAI is for Pytorch what Keras is for Tensorflow: a more intuitive set of abstractions that make it easy to develop deep learning models in a few lines of code.</p> <p>FastAI takes care of a lot of parameters or hyper-parameters for you and it results in good performance by default — everything is customizable of course.</p> <p>But FastAI is not only a library, it is also a great course to get into deep learning. The course is splitted into two parts and is available on <a href="https://course.fast.ai">their website</a> for free. It’s perfect for newcomers as it will let you play quickly with deep learning models without a huge knowledge of the field. This project is entirely based on this course and on FastAI so thanks to them for their awesome work!</p> <p>Now let’s start coding! As usual, we first have to import all the librairies we need through Jupyter:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="n">pandas</span> <span class="k">as</span> <span class="n">pd</span>
<span class="kn">import</span> <span class="n">numpy</span> <span class="k">as</span> <span class="n">np</span>
<span class="kn">import</span> <span class="n">matplotlib.pyplot</span> <span class="k">as</span> <span class="n">plt</span>
<span class="kn">from</span> <span class="n">fastai.vision</span> <span class="kn">import</span> <span class="o">*</span>

<span class="c1"># see matplotlib charts on Jupyter
</span><span class="o">%</span><span class="n">reload_ext</span> <span class="n">autoreload</span>
<span class="o">%</span><span class="n">autoreload</span> <span class="mi">2</span>
<span class="o">%</span><span class="n">matplotlib</span> <span class="n">inline</span>
</code></pre></div></div> <h2 id="pre-processing-the-dataset">Pre-processing the dataset</h2> <p>The first thing we would have to do is pre-processing our dataset. Pre-processing is a very important step, especially when dealing with images. To get better performance or even to let the neural network understand what we are feeding him, we have to <em>normalize</em> the images and resize them.</p> <p>In our project, we are dealing with snapshots that come from movies. Movies can have many different aspect ratios. Think about old movies: most of them were made using a square format — 4/3. But nowadays, it’s all about widescreens — 16/9 — or even wider aspect ratios such as 2.35 or 2.39.</p> <p><img src="/assets/images/2019-07-09-train-multi-label-image-classification/1*wJ7kP6ubpz25IcDIghY8Ow.jpeg" alt="The most-used aspect ratios in the cinema industry."/><em>The most-used aspect ratios in the cinema industry.</em></p> <p>Therefore, to make our AI more efficient, the pre-processing step is crucial: we need to make sure all our datas will have the same format for the training part. To do that, let’s first load our CSV file that contains our dataset:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">path</span> <span class="o">=</span> <span class="nc">Path</span><span class="p">(</span><span class="sh">'</span><span class="s">.</span><span class="sh">'</span><span class="p">)</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">ImageList</span><span class="p">.</span><span class="nf">from_csv</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> 
                          <span class="sh">'</span><span class="s">dataset.csv</span><span class="sh">'</span><span class="p">,</span> 
                          <span class="n">cols</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">folder</span><span class="o">=</span><span class="sh">'</span><span class="s">images</span><span class="sh">'</span><span class="p">,</span> 
                          <span class="n">suffix</span><span class="o">=</span><span class="sh">'</span><span class="s">.jpg</span><span class="sh">'</span><span class="p">)</span>
</code></pre></div></div> <p>With these two lines of code, FastAI basically reads our dataset file and understands that our images are stored in a folder named “images” with the suffix <code class="language-plaintext highlighter-rouge">.jpg</code>. ImageList is a built-in function that lets you do computer vision easily. You can find more informations about this function or about the whole library on their great <a href="https://docs.fast.ai">documentation section</a>.</p> <p>The variable <code class="language-plaintext highlighter-rouge">data</code> now contains everything related to our dataset. We can start the pre-processing:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">data</span> <span class="o">=</span> <span class="n">data</span><span class="p">.</span><span class="nf">split_by_rand_pct</span><span class="p">(</span><span class="mf">0.1</span><span class="p">)</span> <span class="c1"># use 10% of our dataset as validation set
</span>           <span class="p">.</span><span class="nf">label_from_df</span><span class="p">(</span><span class="n">cols</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">label_delim</span><span class="o">=</span><span class="sh">'</span><span class="s">,</span><span class="sh">'</span><span class="p">)</span> <span class="c1"># get the tags
</span>           <span class="p">.</span><span class="nf">transform</span><span class="p">(</span><span class="nf">get_transforms</span><span class="p">(),</span> <span class="n">size</span><span class="o">=</span><span class="mi">224</span><span class="p">,</span> <span class="n">resize_method</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span> <span class="c1"># resize
</span>           <span class="p">.</span><span class="nf">databunch</span><span class="p">(</span><span class="n">bs</span><span class="o">=</span><span class="mi">64</span><span class="p">)</span> <span class="c1"># use mini-batches of size 64
</span>           <span class="p">.</span><span class="nf">normalize</span><span class="p">(</span><span class="n">imagenet_stats</span><span class="p">)</span> <span class="c1"># normalize using the famous imagenet
</span></code></pre></div></div> <p>Most of the comments speak for themselves but I will detail a little bit. The first thing we are doing is use the <code class="language-plaintext highlighter-rouge">split_by_rand_pct</code> method to split the dataset into two parts:</p> <ol> <li> <p><strong>A training set:</strong> It will contain images and tags that will be used to train our AI. We take 90% of the original dataset for the training set.</p> </li> <li> <p><strong>A validation set:</strong> Images and tags that the AI is not using to learn. These images are only seen by the model to validate if it’s learning correctly and the performance is getting better or not. We use 10% of the original dataset for that matter — hence the value “0.1”.</p> </li> </ol> <p>Secondly, we are telling FastAI where to look for the labels of each image with the <code class="language-plaintext highlighter-rouge">label_from_df</code> method. In our case, the labels are located in the second column of our CSV file — not counting the index column — , and are delimited by a comma.</p> <p>Then comes the interesting part, the pre-processing. With the transform method — and more specifically the <code class="language-plaintext highlighter-rouge">get_transforms()</code> function — , we are asking FastAI to perform some work on our images in order to create “more” data. This is called <strong>Data Augmentation</strong>. This is particularly useful when you don’t have enough datas in your training set. FastAI will apply <a href="https://docs.fast.ai/vision.transform.html#get_transforms">by default</a> some techniques to the training images at its disposal, the most common ones being affine transformations — like horizontal or vertical flip, rotation etc.</p> <p>There are also non-affine transformations such as resizing, cropping, brightness variations and more. It will create on each iteration new variant of images based on a single one.</p> <p><a href="/assets/images/2019-07-09-train-multi-label-image-classification/1*3HeYIvxv33bvPRY9uyt2gQ.png"><picture><source sizes="(max-width: 480px) 80vw, (max-width: 979px) 81vw, (max-width: 1199px) 82vw, (min-width: 1200px) 83vw" srcset="/generated/assets/images/2019-07-09-train-multi-label-image-classification/1*3HeYIvxv33bvPRY9uyt2gQ-200-1d4b489ce.webp 200w, /generated/assets/images/2019-07-09-train-multi-label-image-classification/1*3HeYIvxv33bvPRY9uyt2gQ-400-1d4b489ce.webp 400w, /generated/assets/images/2019-07-09-train-multi-label-image-classification/1*3HeYIvxv33bvPRY9uyt2gQ-600-1d4b489ce.webp 600w, /generated/assets/images/2019-07-09-train-multi-label-image-classification/1*3HeYIvxv33bvPRY9uyt2gQ-713-1d4b489ce.webp 713w" type="image/webp"><source sizes="(max-width: 480px) 80vw, (max-width: 979px) 81vw, (max-width: 1199px) 82vw, (min-width: 1200px) 83vw" srcset="/generated/assets/images/2019-07-09-train-multi-label-image-classification/1*3HeYIvxv33bvPRY9uyt2gQ-200-1d4b489ce.png 200w, /generated/assets/images/2019-07-09-train-multi-label-image-classification/1*3HeYIvxv33bvPRY9uyt2gQ-400-1d4b489ce.png 400w, /generated/assets/images/2019-07-09-train-multi-label-image-classification/1*3HeYIvxv33bvPRY9uyt2gQ-600-1d4b489ce.png 600w, /generated/assets/images/2019-07-09-train-multi-label-image-classification/1*3HeYIvxv33bvPRY9uyt2gQ-713-1d4b489ce.png 713w" type="image/png"><img src="/generated/assets/images/2019-07-09-train-multi-label-image-classification/1*3HeYIvxv33bvPRY9uyt2gQ-713-1d4b489ce.png" alt="Examples of data augmentation techniques: rotation, flip, brightness." width="713" height="358"></picture></a><em>Examples of data augmentation techniques: rotation, flip, brightness.</em></p> <p>The next transformation we are doing is the resizing of the images which as you saw, is important in our case. We decided to pick the square size of 224 pixels by 224 pixels because it seems to be a good value for computer vision, especially when normalizing the image with ImageNet which we do on the last line.</p> <p>We decided to pick the <code class="language-plaintext highlighter-rouge">resize_method</code> number “3”. It equals to the “squish” method. The other methods available were “crop”, “pad”, or no resizing at all.</p> <p><img src="/assets/images/2019-07-09-train-multi-label-image-classification/1*hzbBtGkZeoGzeeOP0Ol9Uw.jpeg" alt="The difference methods available to resize our images."/><em>The difference methods available to resize our images.</em></p> <p>We have chosen the “squish” method because after a few tests, we found out that it was the most suitable method in our case considering we wanted to keep all the informations of the original images after the resizing no matter their original size and aspect ratio. It gave us the best results every time.</p> <p>Finally, we normalize our images using the <a href="http://www.image-net.org">ImageNet</a> statistics. Normalization involves using the mean and the standard deviation of the famous ImageNet dataset. Using the mean and the standard deviation is a pretty standard practice. Since they are calculated using a million of images, the statistics are pretty stable. It will make the training faster.</p> <h2 id="define-our-neural-network-structure">Define our neural network structure</h2> <p>Once we are done with the pre-processing, it is time to create the architecture of our neural network and start training our AI. We are going to use <strong>Transfer Learning</strong> for this project. Transfer learning is a technique where a model trained on one task is re-used on a second related task — in our case, detecting elements in movie snapshots. This kind of techniques can be very useful to help, for example, people that don’t have the knowledge to create from scratch their deep neural network neither the ressources or time. Or simply to avoid recoding something entirely when we can use a similar model that works nicely for our type of datas, out of the box.</p> <p>We are going to use one of the most interesting models created recently for image classification called “<a href="https://arxiv.org/abs/1512.03385">Resnet</a>” — the other models available are listed on <a href="https://pytorch.org/docs/stable/torchvision/models.html">Pytorch’s website</a>. Resnet is a residual neural network available in several versions with a depth up to 152 layers — “Resnet152”. It offers incredible results in image recognition.</p> <p>With transfer learning, we are going to take the pre-trained weights of this already trained model — that has been trained on millions of images belonging to 1000 different classes — and use these already learned features to predict new classes — our tags. The main advantage by doing so is that we don’t need an extremely large training dataset to get good results fast.</p> <p>Let’s make our architecture with “Resnet152” as a pre-trained model and create a <a href="https://towardsdatascience.com/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way-3bd2b1164a53">convolutional neural network</a> which is widely used in computer vision. CNN is a Deep Learning algorithm which can take an image as input, assign importance to various aspects or objects in the image and be able to differentiate one from the other. Perfect for us!</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">acc_02</span> <span class="o">=</span> <span class="nf">partial</span><span class="p">(</span><span class="n">accuracy_thresh</span><span class="p">,</span> <span class="n">thresh</span><span class="o">=</span><span class="mf">0.2</span><span class="p">)</span>
<span class="n">f_score</span> <span class="o">=</span> <span class="nf">partial</span><span class="p">(</span><span class="n">fbeta</span><span class="p">,</span> <span class="n">thresh</span><span class="o">=</span><span class="mf">0.2</span><span class="p">)</span>
<span class="n">learn</span> <span class="o">=</span> <span class="nf">cnn_learner</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">models</span><span class="p">.</span><span class="n">resnet152</span><span class="p">,</span> <span class="n">metrics</span><span class="o">=</span><span class="p">[</span><span class="n">acc_02</span><span class="p">,</span><span class="n">f_score</span><span class="p">])</span>
</code></pre></div></div> <p>We defined two functions that will be the metrics of our neural network to help us evaluate if it performs well. The first one is the accuracy threshold. Why a threshold? Simply because here, we are not making a multi-class classifier where we want our AI to predict only one class for every image — the one with the highest probability. We are making a multi-label classifier which means that every image can have different classes — or zero.</p> <p>In our case, we have 183 different classes. So we are going to have one probability for each class for every image. But then, we are not just going to pick out only one of those, we are going to pick out <em>n</em> of those 183 classes. Basically, we compare each probability to some threshold. Then we are going to say that anything that is higher than this threshold means that the image does have that element in it. Jeremy Howard used 0.2 as a threshold for his competition as it seems to generally work pretty well<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>. We decided to follow his advices. Therefore, every class for a given image that have a probability higher than 0.2 will then be considered as part of the image.</p> <p>The second metric is called “FBeta”. It is a well-known metric widely-used in the data scientists industry or in competitions hosted by Kaggle. When you have a classifier, you are going to have some false positives and some false negatives. How do you weigh up those two things to create a single number and evaluate the performance? There are a lots of different ways of doing that and the “FBeta” is a nice way of combining that into a single number. It takes in consideration the precision and the recall. You can read more about the “FBeta” on <a href="https://en.wikipedia.org/wiki/F1_score">Wikipedia</a>.</p> <h2 id="lets-start-training">Let’s start training</h2> <p>We have now a ready-to-use convolutional neural network with Transfer Learning, Resnet152 and two metrics. It’s time to train our data! Let’s start with 10 epochs — one epoch means an iteration over all our images.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">learn</span><span class="p">.</span><span class="nf">fit_one_cycle</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
</code></pre></div></div> <table> <thead> <tr> <th>epoch</th> <th>train_loss</th> <th>valid_loss</th> <th>accuracy_thresh</th> <th>fbeta</th> <th>time</th> </tr> </thead> <tbody> <tr> <td>0</td> <td>0.028299</td> <td>0.025567</td> <td>0.992606</td> <td>0.487419</td> <td>32:35</td> </tr> <tr> <td>1</td> <td>0.026299</td> <td>0.023810</td> <td>0.992710</td> <td>0.529195</td> <td>32:45</td> </tr> <tr> <td>2</td> <td>0.025215</td> <td>0.023037</td> <td>0.992652</td> <td>0.553807</td> <td>32:58</td> </tr> <tr> <td>3</td> <td>0.024489</td> <td>0.022692</td> <td>0.992956</td> <td>0.553749</td> <td>33:15</td> </tr> <tr> <td>4</td> <td>0.023871</td> <td>0.022190</td> <td>0.992828</td> <td>0.570699</td> <td>33:06</td> </tr> <tr> <td>5</td> <td>0.023519</td> <td>0.021935</td> <td>0.992900</td> <td>0.574040</td> <td>32:58</td> </tr> <tr> <td>6</td> <td>0.023003</td> <td>0.021534</td> <td>0.992551</td> <td>0.589838</td> <td>33:06</td> </tr> <tr> <td>7</td> <td>0.023071</td> <td>0.021359</td> <td>0.992833</td> <td>0.590645</td> <td>32:57</td> </tr> <tr> <td>8</td> <td>0.022750</td> <td>0.021231</td> <td>0.992876</td> <td>0.593163</td> <td>32:46</td> </tr> <tr> <td>9</td> <td>0.022044</td> <td>0.021225</td> <td>0.992887</td> <td>0.594099</td> <td>32:53</td> </tr> </tbody> </table> <p>Without that much of work, we get pretty good results already!</p> <p>The accuracy seems fine and the FBeta keeps going up. The Fbeta score might seem low for some of you but remember that we are working over 183 different classes and we are not trying to make a perfect AI that could predict absolutely <em>every</em> element that an image contains. We are trying to make an AI capable of <em>suggesting</em> the most common tags used on the website such as “black and white”, “animation”, “man”, “woman” etc.</p> <p>Still, there is place for improvements. We can try to adjust the learning rate of our neural network to see if it can lead to better results. The learning rate is a very important hyper-parameter in deep learning. If it is chosen poorly, the neural network won’t perform well. FastAI, by default, uses an <a href="https://docs.fast.ai/callbacks.one_cycle.html">one-cycle policy</a> which means the learning rate will first increase over each iteration and then decrease.</p> <p>The one-cycle policy allows to train a neural network very quickly, meaning it will converge fast to a low loss value. But to make it work, we need to find the optimum learning rate that will make the one-cycle policy really shine. FastAI provides functions to find the best learning rate:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">learn</span><span class="p">.</span><span class="nf">unfreeze</span><span class="p">()</span>
<span class="n">learn</span><span class="p">.</span><span class="nf">lr_find</span><span class="p">()</span>
<span class="n">learn</span><span class="p">.</span><span class="n">recorder</span><span class="p">.</span><span class="nf">plot</span><span class="p">(</span><span class="n">suggestion</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
</code></pre></div></div> <p><a href="/assets/images/2019-07-09-train-multi-label-image-classification/1*7wsn-VsP85Q2HCIt8b409A.png"><picture><source sizes="(max-width: 480px) 80vw, (max-width: 979px) 81vw, (max-width: 1199px) 82vw, (min-width: 1200px) 83vw" srcset="/generated/assets/images/2019-07-09-train-multi-label-image-classification/1*7wsn-VsP85Q2HCIt8b409A-200-037c698a3.webp 200w, /generated/assets/images/2019-07-09-train-multi-label-image-classification/1*7wsn-VsP85Q2HCIt8b409A-400-037c698a3.webp 400w, /generated/assets/images/2019-07-09-train-multi-label-image-classification/1*7wsn-VsP85Q2HCIt8b409A-418-037c698a3.webp 418w" type="image/webp"><source sizes="(max-width: 480px) 80vw, (max-width: 979px) 81vw, (max-width: 1199px) 82vw, (min-width: 1200px) 83vw" srcset="/generated/assets/images/2019-07-09-train-multi-label-image-classification/1*7wsn-VsP85Q2HCIt8b409A-200-037c698a3.png 200w, /generated/assets/images/2019-07-09-train-multi-label-image-classification/1*7wsn-VsP85Q2HCIt8b409A-400-037c698a3.png 400w, /generated/assets/images/2019-07-09-train-multi-label-image-classification/1*7wsn-VsP85Q2HCIt8b409A-418-037c698a3.png 418w" type="image/png"><img src="/generated/assets/images/2019-07-09-train-multi-label-image-classification/1*7wsn-VsP85Q2HCIt8b409A-418-037c698a3.png" alt="The loss seems to explode with a large learning rate." width="418" height="262"></picture></a><em>The loss seems to explode with a large learning rate.</em></p> <p>We can see on the chart above that a large learning rate doesn’t seem to help. In fact, the bigger the learning rate is, the bigger the loss is. And we don’t want that. We can restrain our learning rate to avoid this and start again the training with this updated parameter. Also, we are not only going to restart the training with only the last layers of our neural network being updated. Instead, we are going to update this time all the weights, including the Resnet’s 152 layers weights with the function <code class="language-plaintext highlighter-rouge">unfreeze()</code> used above.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">learn</span><span class="p">.</span><span class="nf">fit_one_cycle</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span> <span class="n">max_lr</span><span class="o">=</span><span class="nf">slice</span><span class="p">(</span><span class="mf">1e-06</span><span class="p">,</span><span class="mf">1e-04</span><span class="p">))</span>
</code></pre></div></div> <table> <thead> <tr> <th>epoch</th> <th>train_loss</th> <th>valid_loss</th> <th>accuracy_thresh</th> <th>fbeta</th> <th>time</th> </tr> </thead> <tbody> <tr> <td>0</td> <td>0.022467</td> <td>0.021131</td> <td>0.992844</td> <td>0.596691</td> <td>42:52</td> </tr> <tr> <td>1</td> <td>0.022203</td> <td>0.021018</td> <td>0.992805</td> <td>0.601539</td> <td>40:42</td> </tr> <tr> <td>2</td> <td>0.021539</td> <td>0.020817</td> <td>0.992989</td> <td>0.602677</td> <td>40:57</td> </tr> <tr> <td>3</td> <td>0.021007</td> <td>0.020594</td> <td>0.992868</td> <td>0.612011</td> <td>42:45</td> </tr> <tr> <td>4</td> <td>0.020509</td> <td>0.020398</td> <td>0.992928</td> <td>0.616314</td> <td>40:44</td> </tr> <tr> <td>5</td> <td>0.019788</td> <td>0.020305</td> <td>0.992734</td> <td>0.625013</td> <td>40:57</td> </tr> <tr> <td>6</td> <td>0.018687</td> <td>0.020285</td> <td>0.993009</td> <td>0.622852</td> <td>42:56</td> </tr> <tr> <td>7</td> <td>0.018598</td> <td>0.020303</td> <td>0.992985</td> <td>0.628156</td> <td>40:44</td> </tr> <tr> <td>8</td> <td>0.018053</td> <td>0.020240</td> <td>0.992838</td> <td>0.628797</td> <td>40:59</td> </tr> <tr> <td>9</td> <td>0.018110</td> <td>0.020274</td> <td>0.992997</td> <td>0.627074</td> <td>42:42</td> </tr> </tbody> </table> <p>The train loss seems to get lower and the FBeta seems to get higher as well. Everything looks awesome! We could try another learning rate and train a little more hoping to increase the performance of our model but we can see already that <strong>we are most likely in the optimal capacity that falls between underfitting and overfitting</strong>.</p> <p><a href="/assets/images/2019-07-09-train-multi-label-image-classification/1*2PTV4-P2qNlPCsywgmUsiA-20210330211116341.png"><picture><source sizes="(max-width: 480px) 80vw, (max-width: 979px) 81vw, (max-width: 1199px) 82vw, (min-width: 1200px) 83vw" srcset="/generated/assets/images/2019-07-09-train-multi-label-image-classification/1*2PTV4-P2qNlPCsywgmUsiA-20210330211116341-200-a0a094f55.webp 200w, /generated/assets/images/2019-07-09-train-multi-label-image-classification/1*2PTV4-P2qNlPCsywgmUsiA-20210330211116341-400-a0a094f55.webp 400w, /generated/assets/images/2019-07-09-train-multi-label-image-classification/1*2PTV4-P2qNlPCsywgmUsiA-20210330211116341-600-a0a094f55.webp 600w, /generated/assets/images/2019-07-09-train-multi-label-image-classification/1*2PTV4-P2qNlPCsywgmUsiA-20210330211116341-721-a0a094f55.webp 721w" type="image/webp"><source sizes="(max-width: 480px) 80vw, (max-width: 979px) 81vw, (max-width: 1199px) 82vw, (min-width: 1200px) 83vw" srcset="/generated/assets/images/2019-07-09-train-multi-label-image-classification/1*2PTV4-P2qNlPCsywgmUsiA-20210330211116341-200-a0a094f55.png 200w, /generated/assets/images/2019-07-09-train-multi-label-image-classification/1*2PTV4-P2qNlPCsywgmUsiA-20210330211116341-400-a0a094f55.png 400w, /generated/assets/images/2019-07-09-train-multi-label-image-classification/1*2PTV4-P2qNlPCsywgmUsiA-20210330211116341-600-a0a094f55.png 600w, /generated/assets/images/2019-07-09-train-multi-label-image-classification/1*2PTV4-P2qNlPCsywgmUsiA-20210330211116341-721-a0a094f55.png 721w" type="image/png"><img src="/generated/assets/images/2019-07-09-train-multi-label-image-classification/1*2PTV4-P2qNlPCsywgmUsiA-20210330211116341-721-a0a094f55.png" alt="the optimal capacity that falls between underfitting and overfitting" width="721" height="449"></picture></a><em>The figure shows the optimal capacity that falls between underfitting and overfitting. When reached, we can stop the training. It’s called “early stopping”<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup></em></p> <p>Indeed, after the 4th epoch, our validation loss starts to be higher than the training loss. But more importantly, it doesn’t seem to decrease anymore while in the meantime, the training loss keeps decreasing. We can even see that the validation loss starts to increase on some epoch. This is definitely a sign that we might have reached the <em>optimal capacity</em> of our model and we should stop the training here otherwise we might start overfitting.</p> <p>Overfitting happens when you train too much your model on the same data. When it’s time to test your model on datas it has never seen, out of its comfort zone, the model starts to perform poorly because it only got used to recognize the training datas and nothing else. It does not generalize well on new, unseen data. The model learned patterns specific to the training data, which are irrelevant in other data.</p> <h2 id="predictions-on-test-images">Predictions on test images</h2> <p>It’s better for us to stop the training here and call it a day. We can first save our model with the <code class="language-plaintext highlighter-rouge">save()</code> function and export our model with all the optimized weights and bias:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">learn</span><span class="p">.</span><span class="nf">save</span><span class="p">(</span><span class="sh">'</span><span class="s">res152-movie-stills-after-lr</span><span class="sh">'</span><span class="p">)</span>
<span class="n">learn</span><span class="p">.</span><span class="nf">export</span><span class="p">(</span><span class="sh">'</span><span class="s">movie-stills-tags-ai.pkl</span><span class="sh">'</span><span class="p">)</span>
</code></pre></div></div> <p>We can quickly test on a various number of images if our AI performs correctly on suggesting tags for movie snapshots. We put a hundred of movie stills the AI has never seen in a folder and created a quick Python script to show the predicted tags for each image with their respective probability:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="n">os</span>

<span class="c1"># load our pre-trained model we just exported
</span><span class="n">rest</span> <span class="o">=</span> <span class="nf">load_learner</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="sh">'</span><span class="s">movie-stills-tags-ai.pkl</span><span class="sh">'</span><span class="p">)</span>

<span class="k">for</span> <span class="n">filename</span> <span class="ow">in</span> <span class="n">os</span><span class="p">.</span><span class="nf">listdir</span><span class="p">(</span><span class="n">path</span><span class="o">/</span><span class="sh">'</span><span class="s">test</span><span class="sh">'</span><span class="p">):</span>
  <span class="n">img</span> <span class="o">=</span> <span class="nf">open_image</span><span class="p">(</span><span class="sh">'</span><span class="s">test/</span><span class="sh">'</span> <span class="o">+</span> <span class="n">filename</span><span class="p">)</span>
  <span class="c1"># resize the images the same way a the pre-processing
</span>  <span class="n">img</span> <span class="o">=</span> <span class="n">img</span><span class="p">.</span><span class="nf">apply_tfms</span><span class="p">(</span><span class="n">tfms</span><span class="o">=</span><span class="nf">get_transforms</span><span class="p">()[</span><span class="mi">1</span><span class="p">],</span> <span class="n">size</span><span class="o">=</span><span class="mi">224</span><span class="p">,</span> <span class="n">resize_method</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
  <span class="c1"># get preds for each class and show only the one &gt; to our thres
</span>  <span class="n">preds</span><span class="p">,</span> <span class="n">idx</span><span class="p">,</span> <span class="n">output</span> <span class="o">=</span> <span class="n">rest</span><span class="p">.</span><span class="nf">predict</span><span class="p">(</span><span class="n">img</span><span class="p">)</span>
  <span class="n">d</span> <span class="o">=</span> <span class="nf">dict</span><span class="p">({</span><span class="n">rest</span><span class="p">.</span><span class="n">data</span><span class="p">.</span><span class="n">classes</span><span class="p">[</span><span class="n">i</span><span class="p">]:</span> <span class="nf">round</span><span class="p">(</span><span class="nf">to_np</span><span class="p">(</span><span class="n">p</span><span class="p">)</span><span class="o">*</span><span class="mi">100</span><span class="p">,</span><span class="mi">2</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">p</span> <span class="ow">in</span> <span class="nf">enumerate</span><span class="p">(</span><span class="n">output</span><span class="p">)</span> <span class="k">if</span> <span class="n">p</span> <span class="o">&gt;</span> <span class="mf">0.2</span><span class="p">})</span>
  <span class="n">img</span><span class="p">.</span><span class="nf">show</span><span class="p">(</span><span class="n">title</span><span class="o">=</span><span class="nf">str</span><span class="p">(</span><span class="n">d</span><span class="p">))</span>
</code></pre></div></div> <p><a href="/assets/images/2019-07-09-train-multi-label-image-classification/test-model.png"><picture><source sizes="(max-width: 480px) 80vw, (max-width: 979px) 81vw, (max-width: 1199px) 82vw, (min-width: 1200px) 83vw" srcset="/generated/assets/images/2019-07-09-train-multi-label-image-classification/test-model-200-2b2436728.webp 200w, /generated/assets/images/2019-07-09-train-multi-label-image-classification/test-model-400-2b2436728.webp 400w, /generated/assets/images/2019-07-09-train-multi-label-image-classification/test-model-600-2b2436728.webp 600w, /generated/assets/images/2019-07-09-train-multi-label-image-classification/test-model-797-2b2436728.webp 797w" type="image/webp"><source sizes="(max-width: 480px) 80vw, (max-width: 979px) 81vw, (max-width: 1199px) 82vw, (min-width: 1200px) 83vw" srcset="/generated/assets/images/2019-07-09-train-multi-label-image-classification/test-model-200-2b2436728.png 200w, /generated/assets/images/2019-07-09-train-multi-label-image-classification/test-model-400-2b2436728.png 400w, /generated/assets/images/2019-07-09-train-multi-label-image-classification/test-model-600-2b2436728.png 600w, /generated/assets/images/2019-07-09-train-multi-label-image-classification/test-model-797-2b2436728.png 797w" type="image/png"><img src="/generated/assets/images/2019-07-09-train-multi-label-image-classification/test-model-797-2b2436728.png" alt="A few examples of the testing set" width="797" height="198"></picture></a><em>A few examples of the testing set with the predicted tags and their respective probability.</em></p> <p>It seems to work well! Sure it does not list <em>all</em> the possible elements displayed in the image but that was never our goal. We wanted to create an AI that could suggest common tags to users when they upload movie snapshots and I think it works nicely. Almost two thirds of the time, the AI suggests <em>all</em> possible elements of the image. It doesn’t mean that the rest of the time it doesn’t suggest anything, just not <em>all</em> elements — maybe one or two tags only.</p> <p>As always, there is room for improvement but for a first model, we are happy with the results. It’s important to remember that only tags that have more than 20% of probability of being present according to our AI are displayed here. We want to suggest tags that we are confident there really are inside the image.</p> <h2 id="whats-next">What’s next?</h2> <p>Now that we have done an AI capable of detecting elements in movie snapshots and exported the model with good performance after testing it briefly, how are we going to turn this single file into a service on our website? How are we going to implement the AI on our servers and interact with the current Ruby on Rails backend? That will be the theme of our next article where we will create a REST API in Python with Docker to allow our website to send requests to the AI which will return what tags are present in a given image. Stay tuned!</p> <div class="footnotes" role="doc-endnotes"> <ol> <li id="fn:1" role="doc-endnote"> <p>In the third video of his 2019 course, Jeremy Howard <a href="https://youtu.be/MpZxV6DVsmM?t=2280">explains</a> what a threshold is and why he picked 0.2. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> <li id="fn:2" role="doc-endnote"> <p><a href="https://arxiv.org/abs/1803.09820">A disciplined approach to neural network hyper-parameters</a>: Part 1 – learning rate, batch size, momentum, and weight decay. <a href="#fnref:2" class="reversefootnote" role="doc-backlink">&#8617;</a></p> </li> </ol> </div>]]></content><author><name>Yann Defretin</name></author><category term="AI"/><summary type="html"><![CDATA[After creating and cleaning a dataset from scratch with Pandas, we are now going to train an AI with our data to distinguish several elements in movie snapshots.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://defret.in/assets/images/2019-07-09-train-multi-label-image-classification/1*wJ7kP6ubpz25IcDIghY8Ow.jpeg"/><media:content medium="image" url="https://defret.in/assets/images/2019-07-09-train-multi-label-image-classification/1*wJ7kP6ubpz25IcDIghY8Ow.jpeg" xmlns:media="http://search.yahoo.com/mrss/"/></entry><entry><title type="html">Create a dataset for a multi-label image classification model</title><link href="https://defret.in/log/create-dataset-multilabel-model" rel="alternate" type="text/html" title="Create a dataset for a multi-label image classification model"/><published>2019-05-28T00:00:00+00:00</published><updated>2019-05-28T00:00:00+00:00</updated><id>https://defret.in/log/multilabel-model-create-dataset</id><content type="html" xml:base="https://defret.in/log/create-dataset-multilabel-model"><![CDATA[<p>Created in 2008, <a href="https://whatthemovie.com">Whatthemovie</a> is a unique movie quiz game where people upload movie stills and the others have to guess from what movie the image was taken. In more than ten years, the website have accumulated more than 470 000 images from 30 000 movies uploaded by our awesome users.</p> <p>We favor quality over quantity. Therefore, not all images submitted are accepted. We could have more than a million images today but that was not our priority. The images uploaded must be interesting for the quiz and the quiz itself should stay quite balanced in terms of difficulty and movie genres.</p> <p>The main area looks like this:</p> <p><a href="/assets/images/2019-05-28-multilabel-model-create-dataset/main-area.png"><picture><source sizes="(max-width: 480px) 80vw, (max-width: 979px) 81vw, (max-width: 1199px) 82vw, (min-width: 1200px) 83vw" srcset="/generated/assets/images/2019-05-28-multilabel-model-create-dataset/main-area-200-39c19beb0.webp 200w, /generated/assets/images/2019-05-28-multilabel-model-create-dataset/main-area-400-39c19beb0.webp 400w, /generated/assets/images/2019-05-28-multilabel-model-create-dataset/main-area-600-39c19beb0.webp 600w, /generated/assets/images/2019-05-28-multilabel-model-create-dataset/main-area-731-39c19beb0.webp 731w" type="image/webp"><source sizes="(max-width: 480px) 80vw, (max-width: 979px) 81vw, (max-width: 1199px) 82vw, (min-width: 1200px) 83vw" srcset="/generated/assets/images/2019-05-28-multilabel-model-create-dataset/main-area-200-39c19beb0.png 200w, /generated/assets/images/2019-05-28-multilabel-model-create-dataset/main-area-400-39c19beb0.png 400w, /generated/assets/images/2019-05-28-multilabel-model-create-dataset/main-area-600-39c19beb0.png 600w, /generated/assets/images/2019-05-28-multilabel-model-create-dataset/main-area-731-39c19beb0.png 731w" type="image/png"><img src="/generated/assets/images/2019-05-28-multilabel-model-create-dataset/main-area-731-39c19beb0.png" alt="Can you guess what movie is behind this shot?" width="731" height="613"></picture></a><em>Can you guess what movie is behind this shot?</em></p> <p>You are given an image taken from a movie scene and you have to guess the movie’s title. The flags mean you don’t have to enter the original title to get points — you can also write it in German, French, Spanish etc. We support a lot of languages for alternative titles to let users from all around the world the possibility to play and compete with the rest of the users.</p> <h2 id="improve-the-uploading-stage">Improve the uploading stage</h2> <p>Uploading new movie stills on the website may sound easy: you pick a movie from our database — or add a new one — , you upload your image and you’re good to go. In reality, users have first to take a snapshot <em>by themselves</em> from a movie with any player available, crop the potential black bars of the image and go through the uploading stage described above.</p> <p>On the last step of the uploading stage, we ask our users to add tags that describe what the image contains, what’s inside — does the image show a street, a car etc. It looks relatively easy too but with all the work they have to go through already, most of our uploaders don’t bother adding tags which is unfortunate because <strong>relevant tags are important to filter shots</strong>.</p> <p><a href="/assets/images/2019-05-28-multilabel-model-create-dataset/tags.png"><picture><source sizes="(max-width: 480px) 80vw, (max-width: 979px) 81vw, (max-width: 1199px) 82vw, (min-width: 1200px) 83vw" srcset="/generated/assets/images/2019-05-28-multilabel-model-create-dataset/tags-200-ee623cb0a.webp 200w, /generated/assets/images/2019-05-28-multilabel-model-create-dataset/tags-400-ee623cb0a.webp 400w, /generated/assets/images/2019-05-28-multilabel-model-create-dataset/tags-600-ee623cb0a.webp 600w, /generated/assets/images/2019-05-28-multilabel-model-create-dataset/tags-800-ee623cb0a.webp 800w, /generated/assets/images/2019-05-28-multilabel-model-create-dataset/tags-1000-ee623cb0a.webp 1000w" type="image/webp"><source sizes="(max-width: 480px) 80vw, (max-width: 979px) 81vw, (max-width: 1199px) 82vw, (min-width: 1200px) 83vw" srcset="/generated/assets/images/2019-05-28-multilabel-model-create-dataset/tags-200-ee623cb0a.png 200w, /generated/assets/images/2019-05-28-multilabel-model-create-dataset/tags-400-ee623cb0a.png 400w, /generated/assets/images/2019-05-28-multilabel-model-create-dataset/tags-600-ee623cb0a.png 600w, /generated/assets/images/2019-05-28-multilabel-model-create-dataset/tags-800-ee623cb0a.png 800w, /generated/assets/images/2019-05-28-multilabel-model-create-dataset/tags-1000-ee623cb0a.png 1000w" type="image/png"><img src="/generated/assets/images/2019-05-28-multilabel-model-create-dataset/tags-800-ee623cb0a.png" alt="The tags between the shot" width="1492" height="354"></picture></a><em>You can see the tags below the shot. When you click on one of them — like “b/w” for black and white — you can see all the black and white movie snapshots uploaded on the website.</em></p> <p>Filtering black and white snapshots might not sound interesting but think about other tags like “gore”, “nudity” or “blood”. These tags show obviously sensitive content and some users don’t want to see these images — at least without asking them first. That’s why tagging images is important. If a user forgets to add the “nudity” or “gore” tag on his shot, a user playing from work — a lot are! — could potentially see this image and nobody wants that to happen in an open plan.</p> <p>In fact, we have added a setting on the website to let users choose if they want this kind of snapshots to be displayed automatically or not. But obviously, if a snapshot has not been tagged as sensitive — “gore” or “nudity” — , it will still show up.</p> <p><strong>Which leads us to our project today:</strong> can we suggest tags to users in an automatic way when they upload their snapshots? Can we build an AI capable of detecting relatively common elements in an image to help people tagging their shots? Can we detect nudity or gore material in an image to prevent sensible people to see them?</p> <p>In this first part of our series, we will start by creating a dataset from our database and clean it with Pandas. Then, in the second article, we will start training an AI using FastAI and try to optimize it to get the best results we can when predicting elements in images.</p> <h2 id="from-mariadb-to-pandas">From MariaDB to Pandas</h2> <p>Our website is powered by <a href="https://rubyonrails.org">Ruby on Rails</a> which makes database structure quite easy to create, understand and maintain. In all the datas we have at disposal for this project, we are actually only interested in two tables: the “shots” table and the “tags” table. These tables have all the datas we need. The former has the image ID and its filename. The latter has all tags added to a specific “refID” — in our case, our image ID.</p> <p>There are several ways to export these tables in a format we care about. Pandas, the library we are going to use to prepare the datas to train our AI, can read several file types such as CSV files. That’s fortunate since <a href="https://phpmyadmin.net">PhpMyAdmin</a> has the hability to export tables straight to CSV files.</p> <p>PhpMyAdmin is a nice way to do the conversion, but we prefer to do the export through the command line to have our CSV files directly on the server. Let’s use <a href="https://dev.mysql.com/doc/refman/8.0/en/mysqldump.html">mysqldump</a> to export our two tables to SQL files first:</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>mysqldump <span class="nt">-u</span> root <span class="nt">-p</span> wts shots <span class="o">&gt;</span> shots.sql
Enter password:
<span class="nv">$ </span>mysqldump <span class="nt">-u</span> root <span class="nt">-p</span> wts tags <span class="o">&gt;</span> tags.sql
Enter password:
<span class="nv">$ </span><span class="nb">ls</span> <span class="k">*</span>.sql
shots.sql tags.sql
</code></pre></div></div> <p>Now that we have the tables exported, we can convert them to CSV files using this <a href="https://github.com/bmtgoncalves/mysqldump-to-csv/blob/master/mysqldump_to_csv.py">great Python script</a> made by James Mishra:</p> <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>wget <span class="nt">-q</span> https://raw.githubusercontent.com/bmtgoncalves/mysqldump-to-csv/master/mysqldump_to_csv.py
<span class="nv">$ </span>python mysqldump_to_csv.py shots.sql <span class="o">&gt;</span> shots.csv
<span class="nv">$ </span>python mysqldump_to_csv.py tags.sql <span class="o">&gt;</span> tags.csv
<span class="nv">$ </span><span class="nb">ls</span> <span class="k">*</span>.csv
shots.csv tags.csv
</code></pre></div></div> <p>We are now ready to work with <a href="https://pandas.pydata.org">Pandas</a>!</p> <h2 id="prepare-and-clean-our-datas-with-pandas">Prepare and clean our datas with Pandas</h2> <p>We now have our datas in the good format. We can start using Pandas to explore the datas, keep only the columns of our tables that we are interested in and merge the two tables to get a pretty nice dataset ready to train our AI.</p> <p>The best way to install Pandas and use it is to install <a href="https://www.anaconda.com">Anaconda</a>. It is a great package that ships by default with a lot of librairies used in Deep Learning such as Pandas. You can find how to install it on <a href="http://docs.continuum.io/anaconda/install/">this page</a>.</p> <p>Another advantage to install Anaconda is <a href="https://jupyter.org">Jupyter</a> which is included by default as well. Jupyter lets you do some Python, Pandas and Machine/Deep Learning coding directly in the browser which is quite convenient because you can then create and display, for example, charts of your datas to check what it shows, take better decisions of what to keep etc. Starting Jupyter is easy as typing `jupyter notebook in the terminal once installed.</p> <h3 id="load-the-datas">Load the datas</h3> <p>Let’s start coding! Once we have started Jupyter, we have to import the libairies we need — in this case, Pandas, Numpy and Matplotlib.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="n">pandas</span> <span class="k">as</span> <span class="n">pd</span>
<span class="kn">import</span> <span class="n">numpy</span> <span class="k">as</span> <span class="n">np</span>
<span class="kn">import</span> <span class="n">matplotlib.pyplot</span> <span class="k">as</span> <span class="n">plt</span>

<span class="c1"># show graphs in jupyter
</span><span class="o">%</span><span class="n">reload_ext</span> <span class="n">autoreload</span>
<span class="o">%</span><span class="n">autoreload</span> <span class="mi">2</span>
<span class="o">%</span><span class="n">matplotlib</span> <span class="n">inline</span>

<span class="c1"># display all columns/rows and what's inside
</span><span class="n">pd</span><span class="p">.</span><span class="nf">set_option</span><span class="p">(</span><span class="sh">'</span><span class="s">display.max_rows</span><span class="sh">'</span><span class="p">,</span> <span class="mi">500</span><span class="p">)</span>
<span class="n">pd</span><span class="p">.</span><span class="nf">set_option</span><span class="p">(</span><span class="sh">'</span><span class="s">display.max_columns</span><span class="sh">'</span><span class="p">,</span> <span class="mi">500</span><span class="p">)</span>
<span class="n">pd</span><span class="p">.</span><span class="nf">set_option</span><span class="p">(</span><span class="sh">'</span><span class="s">display.width</span><span class="sh">'</span><span class="p">,</span> <span class="mi">150</span><span class="p">)</span>
</code></pre></div></div> <p>We can now open our CSV files with Pandas and start digging. Let’s open the “tags” table first.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">tags</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="nf">read_csv</span><span class="p">(</span><span class="sh">'</span><span class="s">tags.csv</span><span class="sh">'</span><span class="p">)</span>
<span class="n">tags</span><span class="p">.</span><span class="nf">head</span><span class="p">()</span>
</code></pre></div></div> <table> <thead> <tr> <th>id</th> <th>value</th> <th>ref_id</th> <th>user_id</th> <th>created_at</th> <th>updated_at</th> <th>type</th> </tr> </thead> <tbody> <tr> <td>265</td> <td>pink panties</td> <td>2</td> <td>6847.0</td> <td>2009-05-29 15:00:57</td> <td>2009-05-29 15:00:57</td> <td>ShotTag</td> </tr> <tr> <td>268</td> <td>tennis</td> <td>3</td> <td>6847.0</td> <td>2009-05-29 15:03:33</td> <td>2009-05-29 15:03:33</td> <td>ShotTag</td> </tr> <tr> <td>274</td> <td>casino chip</td> <td>7</td> <td>6847.0</td> <td>2009-05-29 15:07:55</td> <td>2009-05-29 15:07:55</td> <td>ShotTag</td> </tr> <tr> <td>277</td> <td>wooden rabbit</td> <td>8</td> <td>6847.0</td> <td>2009-05-29 15:08:54</td> <td>2009-05-29 15:08:54</td> <td>ShotTag</td> </tr> <tr> <td>282</td> <td>blindfolded man</td> <td>9</td> <td>6847.0</td> <td>2009-05-29 15:10:12</td> <td>2009-05-29 15:10:12</td> <td>ShotTag</td> </tr> </tbody> </table> <p><em>With <code class="language-plaintext highlighter-rouge">.head()</code>, we can see the first rows of our table and its structure directly in Jupyter.</em></p> <p>The first thing we can do here is to remove the columns we don’t need. The “tags” table contains the <code class="language-plaintext highlighter-rouge">value</code> of the tag, the ID of the shot the tag is linked to — <code class="language-plaintext highlighter-rouge">ref_id</code> — and the user who added it. The columns <code class="language-plaintext highlighter-rouge">created_at</code> and <code class="language-plaintext highlighter-rouge">updated_at</code> are automatically generated by Ruby on Rails when you create new tables through their generators.</p> <p>Let’s only keep the <code class="language-plaintext highlighter-rouge">id</code>, <code class="language-plaintext highlighter-rouge">value</code> and <code class="language-plaintext highlighter-rouge">ref_id</code> columns:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">tags</span><span class="p">.</span><span class="nf">drop</span><span class="p">([</span><span class="sh">'</span><span class="s">user_id</span><span class="sh">'</span><span class="p">,</span> <span class="sh">'</span><span class="s">created_at</span><span class="sh">'</span><span class="p">,</span> <span class="sh">'</span><span class="s">updated_at</span><span class="sh">'</span><span class="p">,</span> <span class="sh">'</span><span class="s">type</span><span class="sh">'</span><span class="p">],</span> <span class="n">axis</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">inplace</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">tags</span><span class="p">.</span><span class="nf">set_index</span><span class="p">(</span><span class="sh">'</span><span class="s">id</span><span class="sh">'</span><span class="p">,</span> <span class="n">inplace</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">tags</span><span class="p">.</span><span class="nf">head</span><span class="p">()</span>
</code></pre></div></div> <table> <thead> <tr> <th>id</th> <th>value</th> <th>ref_id</th> </tr> </thead> <tbody> <tr> <td>265</td> <td>pink panties</td> <td>2</td> </tr> <tr> <td>268</td> <td>tennis</td> <td>3</td> </tr> <tr> <td>274</td> <td>casino chip</td> <td>7</td> </tr> <tr> <td>277</td> <td>wooden rabbit</td> <td>8</td> </tr> <tr> <td>282</td> <td>blindfolded man</td> <td>9</td> </tr> </tbody> </table> <p><em>Only keep the values we care about!</em></p> <p>Our goal is in this project is not to find <em>every</em> possible element in an image but to identify the most common elements or, to be more precise, the most used tags on the website. We can find that quite simply with Pandas and the <code class="language-plaintext highlighter-rouge">.value_counts()</code> function:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">top_tags</span> <span class="o">=</span> <span class="n">tags</span><span class="p">.</span><span class="n">value</span><span class="p">.</span><span class="nf">value_counts</span><span class="p">().</span><span class="n">iloc</span><span class="p">[:</span><span class="mi">400</span><span class="p">]</span>
<span class="n">pd</span><span class="p">.</span><span class="nc">DataFrame</span><span class="p">(</span><span class="n">top_tags</span><span class="p">)</span>
</code></pre></div></div> <table> <thead> <tr> <th style="text-align: center">rank</th> <th>tag</th> <th>count</th> </tr> </thead> <tbody> <tr> <td style="text-align: center">1.</td> <td>b/w</td> <td>41 903</td> </tr> <tr> <td style="text-align: center">2.</td> <td>animation</td> <td>16 865</td> </tr> <tr> <td style="text-align: center">3.</td> <td>text</td> <td>12 638</td> </tr> <tr> <td style="text-align: center">4.</td> <td>nudity</td> <td>8882</td> </tr> <tr> <td style="text-align: center">5.</td> <td>Awesome Upload</td> <td>8634</td> </tr> <tr> <td style="text-align: center">6.</td> <td>car</td> <td>7036</td> </tr> <tr> <td style="text-align: center">7.</td> <td>character shot</td> <td>5227</td> </tr> <tr> <td style="text-align: center">8.</td> <td>gore</td> <td>5096</td> </tr> <tr> <td style="text-align: center">9.</td> <td>woman</td> <td>4671</td> </tr> <tr> <td style="text-align: center">10.</td> <td>man</td> <td>4639</td> </tr> </tbody> </table> <p><em>Some of the most-used tags on the website.</em></p> <p>Showing the top 400 tags used here will be overkill but we can already find some datas to clean in these first results. We can see that in the 5th position, we have the tag “Awesome Upload” or, in the 13th position, we have the “SotD” tag. These tags are useless in our case because there are tags that don’t describe what’s on the image. In fact, they are related to the website’s system.</p> <p>A shot with a “Awesome Upload” tag means it was uploaded by a premium user and in that case, the image is automatically accepted, there is no vote, no selection. Therefore, we don’t want to keep this tag and try to suggest it on a new uploaded image — it is already done automatically.</p> <p>The same applies to the “SotD” tag — which means “Snapshot of the Day”. Every day, the best rated shot gets rewarded, promoted, and the uploader as well. It doesn’t describe anything in the snapshot so we don’t want it.</p> <p>Another interesting one is the “couple” tag. A couple is very hard to recognize. It could be two men, two girls, a man and a girl etc. Also, how and when do we decide to apply this tag? When people are close to each other? Kissing? Hugging? It’s hard to see the difference.</p> <p><strong>More importantly, since our website is a quiz game, it’s important that tags must only describe what the image shows. Tags should not give away any hint about the movie we are trying to guess.</strong> If an uploader — who obviously has watched the movie — adds the “couple” tag, it gives away some information about the movie and we don’t want that. If we see two people kissing, it doesn’t mean they are a couple, neither if they are close to each other.</p> <h3 id="get-rid-of-tags-we-dont-want-to-suggest"><strong>Get rid of tags we don’t want to suggest</strong></h3> <p>Before removing tags, we can merge tags that are very similar. For example, we have shots with the “phone” tag or the “telephone” one. Since it describes the same thing, we can merge them into a single one, “telephone”:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">tags</span><span class="p">[</span><span class="sh">'</span><span class="s">value</span><span class="sh">'</span><span class="p">]</span> <span class="o">=</span> <span class="n">tags</span><span class="p">[</span><span class="sh">'</span><span class="s">value</span><span class="sh">'</span><span class="p">].</span><span class="nf">apply</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">x</span> <span class="o">==</span> <span class="sh">'</span><span class="s">phone</span><span class="sh">'</span> <span class="k">else</span> <span class="sh">'</span><span class="s">telephone</span><span class="sh">'</span><span class="p">)</span>
</code></pre></div></div> <p>We also do the same process for other tags such as “airplane” and “plane” or “redhead” and “red hair” and we drop the potential duplicates found in the table afterwards to be sure everything is clean:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">tags</span><span class="p">.</span><span class="nf">drop_duplicates</span><span class="p">([</span><span class="sh">'</span><span class="s">value</span><span class="sh">'</span><span class="p">,</span> <span class="sh">'</span><span class="s">ref_id</span><span class="sh">'</span><span class="p">],</span> <span class="n">inplace</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
</code></pre></div></div> <p>For this project, we decided to keep tags that were at least added more than 500 times and exclude some tags like the ones described above. Pandas provides a convenient function to check if a column’s value is part of a list of different values — and keep only these ones. Here, we want the opposite. We want to exclude a list of tags from our dataset. We can use the <code class="language-plaintext highlighter-rouge">.isin()</code> function that normally gives back only the rows where the value matches one of the elements of a list. But if we add <code class="language-plaintext highlighter-rouge">~</code> in front of the function, it will do what we are looking for, exclude a list of tags — like a <code class="language-plaintext highlighter-rouge">isNOTin()</code> function.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">top_tags_name</span> <span class="o">=</span> <span class="n">top_tags</span><span class="p">[</span><span class="n">top_tags</span> <span class="o">&gt;=</span> <span class="mi">500</span><span class="p">].</span><span class="n">index</span>
<span class="n">exclude_tags</span> <span class="o">=</span> <span class="p">[</span><span class="sh">'</span><span class="s">Awesome Upload</span><span class="sh">'</span><span class="p">,</span> <span class="sh">'</span><span class="s">SotD</span><span class="sh">'</span><span class="p">,</span> <span class="sh">'</span><span class="s">SotM</span><span class="sh">'</span><span class="p">,</span> <span class="sh">'</span><span class="s">SotY</span><span class="sh">'</span><span class="p">,</span> 
                <span class="sh">'</span><span class="s">replaceplease</span><span class="sh">'</span><span class="p">,</span> <span class="sh">'</span><span class="s">unsolved</span><span class="sh">'</span><span class="p">,</span> <span class="sh">"</span><span class="s">Zo0</span><span class="sh">'</span><span class="s">s eye series</span><span class="sh">"</span><span class="p">,</span> 
                <span class="sh">'</span><span class="s">not that one</span><span class="sh">'</span><span class="p">,</span> <span class="sh">'</span><span class="s">upside down</span><span class="sh">'</span><span class="p">,</span> <span class="sh">'</span><span class="s">hanging</span><span class="sh">'</span><span class="p">,</span> 
                <span class="sh">'</span><span class="s">couple</span><span class="sh">'</span><span class="p">,</span> <span class="sh">'</span><span class="s">close-up</span><span class="sh">'</span><span class="p">,</span> <span class="sh">'</span><span class="s">animated</span><span class="sh">'</span><span class="p">]</span>
<span class="n">top_tags_name_cleaned</span> <span class="o">=</span> <span class="n">top_tags_name</span><span class="p">[</span><span class="o">~</span><span class="n">top_tags_name</span><span class="p">.</span><span class="nf">isin</span><span class="p">(</span><span class="n">exclude_tags</span><span class="p">)]</span>
</code></pre></div></div> <p>After this operation, we have 183 different tags left. We can now keep only the rows in our “tags” table that contain one of them with the <code class="language-plaintext highlighter-rouge">.isin()</code> function:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">tags_cleaned</span> <span class="o">=</span> <span class="n">tags</span><span class="p">[</span><span class="n">tags</span><span class="p">[</span><span class="sh">'</span><span class="s">value</span><span class="sh">'</span><span class="p">].</span><span class="nf">isin</span><span class="p">(</span><span class="nf">list</span><span class="p">(</span><span class="n">top_tags_name_cleaned</span><span class="p">.</span><span class="n">values</span><span class="p">))]</span>
</code></pre></div></div> <p>Our “tags” table is now ready to be merged!</p> <h2 id="merge-the-tags-table-with-the-shots-table">Merge the “tags” table with the “shots” table</h2> <p>Now that we have cleaned our “tags” table and kept only the most-used and interesting tags, we have to use Pandas to create a unique table that contains the tags and the image datas. We know that the <code class="language-plaintext highlighter-rouge">ref_id</code> column in the “tags” table refers to the the <code class="language-plaintext highlighter-rouge">id</code> of an image in the “shots” table. Pandas makes it easy to merge two different tables that have a “key” in common with the merge function.</p> <p>Let’s first load and check the structure of our “shots” table:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">shots</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="nf">read_csv</span><span class="p">(</span><span class="sh">"</span><span class="s">shots.csv</span><span class="sh">"</span><span class="p">)</span>
<span class="c1"># only keep columns we care about
</span><span class="n">shots</span> <span class="o">=</span> <span class="n">shots</span><span class="p">[[</span><span class="sh">'</span><span class="s">id</span><span class="sh">'</span><span class="p">,</span> <span class="sh">'</span><span class="s">movie_id</span><span class="sh">'</span><span class="p">,</span> <span class="sh">'</span><span class="s">queue</span><span class="sh">'</span><span class="p">,</span> <span class="sh">'</span><span class="s">shot_image_id</span><span class="sh">'</span><span class="p">]]</span>
<span class="n">shots</span><span class="p">.</span><span class="nf">head</span><span class="p">()</span>
</code></pre></div></div> <table> <thead> <tr> <th style="text-align: center">id</th> <th style="text-align: center">movie_id</th> <th style="text-align: center">queue</th> <th style="text-align: center">shot_image_id</th> </tr> </thead> <tbody> <tr> <td style="text-align: center">1</td> <td style="text-align: center">1</td> <td style="text-align: center">3</td> <td style="text-align: center">1</td> </tr> <tr> <td style="text-align: center">2</td> <td style="text-align: center">2</td> <td style="text-align: center">3</td> <td style="text-align: center">81091</td> </tr> <tr> <td style="text-align: center">3</td> <td style="text-align: center">3</td> <td style="text-align: center">3</td> <td style="text-align: center">81021</td> </tr> <tr> <td style="text-align: center">4</td> <td style="text-align: center">4</td> <td style="text-align: center">3</td> <td style="text-align: center">80993</td> </tr> <tr> <td style="text-align: center">6</td> <td style="text-align: center">6</td> <td style="text-align: center">3</td> <td style="text-align: center">82384</td> </tr> </tbody> </table> <p><em>A few columns of our “shots” table.</em></p> <p>The only columns we care about here are the <code class="language-plaintext highlighter-rouge">id</code> column — which refers to the shot ID on the website — , and the <code class="language-plaintext highlighter-rouge">shot_image_id</code> column that stores the filename of the movie snapshot. It’s time to merge both tables! As we know already, we have a key in common between these two tables so it’s going to be easy.</p> <p>Let’s use Pandas to do the work:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">merged_data</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="nf">merge</span><span class="p">(</span><span class="n">shots</span><span class="p">,</span> <span class="n">tags_cleaned</span><span class="p">,</span> <span class="n">left_on</span><span class="o">=</span><span class="p">[</span><span class="sh">'</span><span class="s">id</span><span class="sh">'</span><span class="p">],</span> <span class="n">right_on</span><span class="o">=</span><span class="p">[</span><span class="sh">'</span><span class="s">ref_id</span><span class="sh">'</span><span class="p">])</span>
<span class="n">merged_data</span><span class="p">.</span><span class="nf">head</span><span class="p">()</span>
</code></pre></div></div> <table> <thead> <tr> <th>id</th> <th>movie_id</th> <th>queue</th> <th>shot_image_id</th> <th>value</th> <th>ref_id</th> </tr> </thead> <tbody> <tr> <td>1</td> <td>1</td> <td>3</td> <td>1</td> <td>statue</td> <td>1</td> </tr> <tr> <td>2</td> <td>2</td> <td>3</td> <td>81091</td> <td>bed</td> <td>2</td> </tr> <tr> <td>2</td> <td>2</td> <td>3</td> <td>81091</td> <td>legs</td> <td>2</td> </tr> <tr> <td>2</td> <td>2</td> <td>3</td> <td>81091</td> <td>text</td> <td>2</td> </tr> <tr> <td>7</td> <td>7</td> <td>3</td> <td>250992</td> <td>hand</td> <td>7</td> </tr> </tbody> </table> <p><em>Both tables have been merged into one.</em></p> <p>After merging the datas, we are going, once again, to remove all the columns we are not interested in for our final dataset. But we’re also going to regroup all the tags into a single cell for each image. For example, you can see above that the image with the <code class="language-plaintext highlighter-rouge">id</code> “2” have three different rows meaning it has three different tags associated to it — “bed”, “legs”, “text” in the <code class="language-plaintext highlighter-rouge">value</code> column.</p> <p>We’re going to regroup all the tags for a given image in a single cell separated by a comma:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># regroup tags for a given image into one cell
</span><span class="n">tags_grouped</span> <span class="o">=</span> <span class="n">merged_data</span><span class="p">.</span><span class="nf">groupby</span><span class="p">([</span><span class="sh">'</span><span class="s">ref_id</span><span class="sh">'</span><span class="p">,</span> <span class="sh">'</span><span class="s">shot_image_id</span><span class="sh">'</span><span class="p">])[</span><span class="sh">'</span><span class="s">value</span><span class="sh">'</span><span class="p">].</span><span class="nf">apply</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="sh">'</span><span class="s">,</span><span class="sh">'</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="nf">list</span><span class="p">(</span><span class="n">x</span><span class="p">)))</span>

<span class="c1"># change table's index
</span><span class="n">tags_grouped</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="nc">DataFrame</span><span class="p">(</span><span class="n">tags_grouped</span><span class="p">).</span><span class="nf">reset_index</span><span class="p">()</span>
<span class="n">tags_grouped</span><span class="p">.</span><span class="nf">set_index</span><span class="p">(</span><span class="sh">'</span><span class="s">ref_id</span><span class="sh">'</span><span class="p">,</span> <span class="n">inplace</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">tags_grouped</span><span class="p">.</span><span class="n">index</span><span class="p">.</span><span class="n">names</span> <span class="o">=</span> <span class="p">[</span><span class="sh">'</span><span class="s">id</span><span class="sh">'</span><span class="p">]</span>

<span class="c1"># rename columns
</span><span class="n">tags_grouped</span><span class="p">.</span><span class="n">columns</span> <span class="o">=</span> <span class="p">[</span><span class="sh">'</span><span class="s">shot_image_id</span><span class="sh">'</span><span class="p">,</span> <span class="sh">'</span><span class="s">tags</span><span class="sh">'</span><span class="p">]</span>
<span class="n">tags_grouped</span><span class="p">.</span><span class="nf">head</span><span class="p">()</span>
</code></pre></div></div> <table> <thead> <tr> <th>id</th> <th>shot_image_id</th> <th>tags</th> </tr> </thead> <tbody> <tr> <td>1</td> <td>1</td> <td>statue</td> </tr> <tr> <td>2</td> <td>81091</td> <td>bed, legs, text</td> </tr> <tr> <td>7</td> <td>250992</td> <td>hand, text</td> </tr> <tr> <td>8</td> <td>81360</td> <td>castle, sky</td> </tr> <tr> <td>10</td> <td>9</td> <td>water, redhead</td> </tr> </tbody> </table> <p><em>Our final table: an image ID and its tags.</em></p> <p>For clarity purpose, we also have renamed the column containing the tags to <code class="language-plaintext highlighter-rouge">tags</code>, set the column <code class="language-plaintext highlighter-rouge">ref_id</code> as index and renamed it to <code class="language-plaintext highlighter-rouge">id</code>. Good, now our dataset is ready to train our AI! By using <code class="language-plaintext highlighter-rouge">.shape</code>, we see that we have 243 558 images in our dataset with at least one associated tag to it and 183 different labels.</p> <p>The last thing to do is to export our dataset to a CSV file:</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">tags_grouped</span><span class="p">.</span><span class="nf">to_csv</span><span class="p">(</span><span class="sh">'</span><span class="s">dataset.csv</span><span class="sh">'</span><span class="p">)</span>
</code></pre></div></div> <h2 id="whats-next">What’s next</h2> <p>In this first article of our series, we created our own dataset from scratch using Pandas. After cleaning it and keeping only useful datas, it is now time to start thinking about how to use this dataset to create and train an AI capable of distinguishing different elements in movie snapshots and suggest these elements as tags to users. That will be the theme of our second article. Stay tuned!</p>]]></content><author><name>Yann Defretin</name></author><category term="Pandas"/><summary type="html"><![CDATA[We are going to make an AI capable of distinguishing several elements in movie stills by first using Pandas to create our dataset.]]></summary><media:thumbnail xmlns:media="http://search.yahoo.com/mrss/" url="https://defret.in/assets/images/2019-05-28-multilabel-model-create-dataset/main-area.png"/><media:content medium="image" url="https://defret.in/assets/images/2019-05-28-multilabel-model-create-dataset/main-area.png" xmlns:media="http://search.yahoo.com/mrss/"/></entry></feed>