{"id":860,"date":"2025-07-07T21:45:53","date_gmt":"2025-07-07T21:45:53","guid":{"rendered":"https:\/\/cristianku.com\/?p=860"},"modified":"2025-07-07T21:55:08","modified_gmt":"2025-07-07T21:55:08","slug":"deep-learning-training-with-huge-amount-of-data-part2","status":"publish","type":"post","link":"https:\/\/cristianku.com\/it\/2025\/07\/07\/deep-learning-training-with-huge-amount-of-data-part2\/","title":{"rendered":"Deep Learning . Training with huge amount of data .PART2"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

If you read the first part we left at the point that we are goint to build a multi-thread, multiprocesses mechanism.<\/p>

CPU1: <\/strong>reads a big chunk from disk ( 3200 samples ), shuffle it, and insert a memory Buffer ( Queue)<\/p>

CPU2: <\/strong>the second one is the generator <\/strong>that extract a small amount from this buffer ( the batch size = 32 samples)<\/p>

GPU: last one is the Keras\/Tensorflow calculations done in the GPU<\/strong><\/p>

For this purpose we do use the Python Queue:<\/strong><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\r\n\t\t The Queue module implements multi-producer, multi-consumer queues.\nIt is especially useful in threaded programming when information must be exchanged safely between multiple threads.\nThe Queue class in this module implements all the required locking semantics.\nIt depends on the availability of thread support in Python; see the threading module.\t\t\t\t\t<\/blockquote>\r\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Let\u2019s define the queues:<\/p>

<\/figure>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

One important thing to note about the Python Queue <\/strong>, is , that defining the max size<\/em>, we can decide HOW MANY IMAGES<\/strong>, put in memory.<\/p>

And the instruction <Queue name>.put<\/strong> will put data until the Queue is full and then wait until the Queue will be have free space.<\/p>

In our case the put method will be wait for Tensorflow to get the data<\/p>

First we need a Function that can be put in a threaded process :<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

And the generator function will become now very short :<\/p>

<\/figure>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

This because the dirty work is done by the Queue loader \u2192 read_images_into_queue. <\/strong>In the generator we need only to send back to Tensorflow that Samples, as many as the batch_size<\/strong><\/p>

We will now create two threads ( processes )<\/strong><\/p>

The first for the Training samples<\/strong> and the second<\/strong> for the Validation Samples<\/strong>. They are completely separated from each other:<\/p>

<\/figure>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Remember to terminate the producers ( processes ) when you finish to use them , because if you remember the Queue Loader has a infinite loop ( while 1: ).<\/strong><\/p>

Now we are going to create the generators objects, normally, just using the Queues instead of the Pytables ( or instead of numpy arrays )<\/p>

<\/figure>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Last step: Train the Network<\/strong><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

As you can see below we are using here two cores and the GPU is all the time around 70%<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Here the training statistic:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
Epoch 1\/1
2807\/2806 [==============================] - 29s - loss: 0.0368 - val_loss: 0.0271

Total number of train samples: 89820 ( shape 128x128)

Batch Size                   : 32

Duration                     : 0:00:29.998415
  
 .. model saved to selfdrive_model.h5<\/span><\/pre>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
And\u2026 yes because the Intel i3 that I am using is Hyperthreading.. it nicely splits the work between the cores\u2026 But its also thanks all the asynchronous process that we have built that the each core usage is between 40\u201350 % !<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
ValueError: When creating EArrays, you need to set one of the dimensions of the Atom instance to zero.<\/pre>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
Remember to terminate the Queue Loaders !:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
training_producer.terminate()
validation_producer.terminate()<\/strong><\/pre>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"
If you read the first part we left at the point that we are goint to build a multi-thread, multiprocesses mechanism. CPU1: reads a big chunk from disk ( 3200 samples ), shuffle it, and insert a memory Buffer ( Queue) CPU2: the second one is the generator that extract a small amount from this…<\/p>\n
Read More<\/a><\/p>","protected":false},"author":1,"featured_media":640,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[38],"tags":[35,34,36,37],"class_list":["post-860","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-deep-learning","tag-ai","tag-deep-learning","tag-neural-networks","tag-python"],"jetpack_featured_media_url":"https:\/\/cristianku.com\/wp-content\/uploads\/2023\/02\/Screenshot-2025-07-07-at-10.54.15-scaled.jpg","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/cristianku.com\/it\/wp-json\/wp\/v2\/posts\/860","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cristianku.com\/it\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cristianku.com\/it\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cristianku.com\/it\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/cristianku.com\/it\/wp-json\/wp\/v2\/comments?post=860"}],"version-history":[{"count":10,"href":"https:\/\/cristianku.com\/it\/wp-json\/wp\/v2\/posts\/860\/revisions"}],"predecessor-version":[{"id":878,"href":"https:\/\/cristianku.com\/it\/wp-json\/wp\/v2\/posts\/860\/revisions\/878"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cristianku.com\/it\/wp-json\/wp\/v2\/media\/640"}],"wp:attachment":[{"href":"https:\/\/cristianku.com\/it\/wp-json\/wp\/v2\/media?parent=860"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cristianku.com\/it\/wp-json\/wp\/v2\/categories?post=860"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cristianku.com\/it\/wp-json\/wp\/v2\/tags?post=860"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}