{"id":234,"date":"2023-02-02T11:01:09","date_gmt":"2023-02-02T11:01:09","guid":{"rendered":"https:\/\/cristianku.com\/?p=234"},"modified":"2025-07-07T17:34:09","modified_gmt":"2025-07-07T17:34:09","slug":"deep-learning-huge-amount-1","status":"publish","type":"post","link":"https:\/\/cristianku.com\/it\/2023\/02\/02\/deep-learning-huge-amount-1\/","title":{"rendered":"Deep Learning . Training with huge amount of data .PART1"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Performance considerations
I would like to put some consideration about Performance in Keras , in order to obtain the max GPU consideration.
What we are going to do , is train a Keras Model.
Here the number of images :
Now we define a generator used to train in batch of 32 ( can be an other number ), and we put this generator in the following call:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\r\n\t\t model.fit_generator(train_generator, steps_per_epoch= \\\nnumper_of_train_samples\/batch_size, validation_data=validation_generator, \\\nvalidation_steps=number_of_validation_samples\/batch_size, epochs=epochs, verbose = 1)\t\t\t\t\t<\/blockquote>\r\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The generator is so defined that reads image from disk, n-images exactly how the batch size is:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t
\r\n\t\t def generator(generator_samples, batch_size, shape ):\nnum_samples = len(generator_samples)\nwhile 1: # Loop forever so the generator never terminates\n\u2026 read from disk and append info images and angles array\n# end of the batch, yield the images gathered\nyield sklearn.utils.shuffle(np.array(images) , np.array(angles))\t\t\t\t\t<\/blockquote>\r\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Now we start the fit_generator\u2026 and take a closer look at the CPU and GPU utilization to understand how the data is flushing in between , from CPU to GPU and back\u2026.. Look at the CPU working at full speed in order to handle the sparse disk reads and the GPU waiting for job\u2026 and sometimes doing nothing<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Training Statistic:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

So .. whats the next idea ?<\/p>\n

Very simple ! load ALL the images in MEMORY !!<\/p>\n

Is a good idea ?? Yes and now\u2026 if we have a lot of memory\u2026 but its not a scalable option because if we add in the future new images.. it can happen that the program explodes\u2026in out of memory exception\u2026<\/p>\n

but lets take a look at the performance now :<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

As you can see the GPU is at full speed. Although it was not a full speed all the time. We see later to resolve this with a threaded generator.
IS FITTING EVERYTHING IN MEMORY A GOOD IDEA ??
The answer is : NO!!!!
Ideally we should try to keep a buffer in memory where a \u201cworker\u201d can feed the GPU very fast, and .. the remaining data on a SSD waiting to be put in the Memory buffer by an other \u201cworker\u201d
Lets give a try to the Pytables, saved in a .hdf5 file.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
def generator(samples, labels, batch_size, shape ):
<\/strong>    num_samples = len(samples)
    while 1: # Loop forever so the generator never terminates
#         sklearn.utils.shuffle(samples)
        for offset in range(0, num_samples, batch_size):
            batch_samples = samples[offset:offset+batch_size]
            batch_labels  = labels[offset:offset+batch_size]<\/span>images = []
            angles = []<\/mark>
            for batch_sample, batch_label in zip(batch_samples, batch_labels):
                
                images.append(batch_sample)
                angles.append(batch_label)<\/span># end of the batch, yield the images gathered
#             print (batch_sample.shape)
            yield sklearn.utils.shuffle(np.array(images) , np.array(angles))
#             return sklearn.utils.shuffle(np.array(images)  , np.array(angles))<\/span>
py_training_samples.append(output[0][None]<\/strong>)<\/span><\/pre>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
The None is needed to creare a additional first dimension, as the output[0] is the image size, so we end to have a (1,128,128,3 ) that is the same as defined above: shape=( 0,resized_shape, resized_shape, 3))
If you try to define the table fieldshape=( resized_shape, resized_shape, 3)), without the first dimension, you will get this error:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
ValueError: When creating EArrays, you need to set one of the dimensions of the Atom instance to zero.<\/pre>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
What about the file dimension ? , 5 Gigabytes!!!!:
-rw-r \u2014 r \u2014 1 cristianku cristianku 4.993.236.504 Oct 9 23:51 samples.hdf5
Now lets configure the generator to use the Pytable \u2026 No! there is no adjustment here.. it works exactly like a Numpy Array !:
Here my generator:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
def generator(samples, labels, batch_size, shape ):
    num_samples = len(samples)
    while 1: # Loop forever so the generator never terminates
#         sklearn.utils.shuffle(samples)
        for offset in range(0, num_samples, batch_size):
            batch_samples = samples[offset:offset+batch_size]
            batch_labels  = labels[offset:offset+batch_size]<\/span>images = []
            angles = []
            for batch_sample, batch_label in zip(batch_samples, batch_labels):
                
                images.append(batch_sample)
                angles.append(batch_label)<\/span># end of the batch, yield the images gathered
#             print (batch_sample.shape)
            yield sklearn.utils.shuffle(np.array(images) , np.array(angles))
#             return sklearn.utils.shuffle(np.array(images)  , np.array(angles))<\/span><\/pre>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
Epoch 1\/1
2541\/2540 [==============================] - 35s - loss: 0.0679 - val_loss: 0.0407

Total number of train samples: 81282 ( shape 128x128)

Batch Size : 32

Duration : 0:00:35.566706<\/pre>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
And now the verdict ! The Performance ???
Its even better than the entire numpy array in memory !!!<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
We still have a problem.
Normally , for each epoch, we do Shuffle the data , in order to have more \u201crandomly distributed training data\u201d
how we can shuffle 5 giga of data ??
So\u2026 we have a batch size of 32 sample for example\u2026 we can read from Pytable more than that in a single fetch.. lets say 10 times more\u2026 and shuffle this chuck of data:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
def generator(samples, labels, batch_size, shape ):
 num_samples = len(samples)
 while 1: # Loop forever so the generator never terminates
# sklearn.utils.shuffle(samples)
 for offset in range(0, num_samples, batch_size*100):
 # loading into memory a BIG chunk of data ( 3200 samples )
 chunk_batch_samples = samples[offset:offset+batch_size*100]
 chunk_batch_labels = labels[offset:offset+batch_size*100]
 # shuffling them 
 chunk_batch_samples,chunk_batch_labels = sklearn.utils.shuffle(chunk_batch_samples,chunk_batch_labels)
 
# print (\u201c offset = {}, len(chunk_batch_samples) {}\u201d.format(offset, len(chunk_batch_samples)))
 images = []
 angles = []
 # looping on the Chunk of 3200 samples , extracting 32 elements ( batch_size) every time
 for offset2 in range(0, len(chunk_batch_samples), batch_size):
 chunk_batch_samples = chunk_batch_samples[offset2:offset2+batch_size]
 chunk_batch_labels = chunk_batch_labels[offset2:offset2+batch_size]<\/span>for batch_sample, batch_label in zip(chunk_batch_samples, chunk_batch_labels):<\/span>images.append(batch_sample)
 angles.append(batch_label)<\/span>yield sklearn.utils.shuffle(np.array(images) , np.array(angles))<\/span><\/pre>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
With this method, we do Shuffle the data, maintain a Memory at only 25%<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
and we have more or less the same good performance of having everything in memory:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t
Our Training is NOW SCALABLE, FAST, and RELIABLE is indipendent from the number of Images to train\u2026 we can have a million of images and the machanism is the same !
Whats next ? Next time I will go deep and further improve the mechanism, by introducing two different threads :
CPU1: reads a big chunk from disk ( 3200 samples ), shuffle it, and insert a memory Buffer ( Queue)
CPU2: the second one is the generator that extract a small amount from this buffer ( the batch size = 32 samples)
GPU: last one is the Keras\/Tensorflow calculations done in the GPU<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"
Performance considerationsI would like to put some consideration about Performance in Keras , in order to obtain the max GPU consideration.What we are going to do , is train a Keras Model.Here the number of images :Now we define a generator used to train in batch of 32 ( can be an other number ),…<\/p>\n
Read More<\/a><\/p>","protected":false},"author":1,"featured_media":640,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[38],"tags":[35,34,36,37],"class_list":["post-234","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-deep-learning","tag-ai","tag-deep-learning","tag-neural-networks","tag-python"],"jetpack_featured_media_url":"https:\/\/cristianku.com\/wp-content\/uploads\/2023\/02\/Screenshot-2025-07-07-at-10.54.15-scaled.jpg","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/cristianku.com\/it\/wp-json\/wp\/v2\/posts\/234","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cristianku.com\/it\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cristianku.com\/it\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cristianku.com\/it\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/cristianku.com\/it\/wp-json\/wp\/v2\/comments?post=234"}],"version-history":[{"count":13,"href":"https:\/\/cristianku.com\/it\/wp-json\/wp\/v2\/posts\/234\/revisions"}],"predecessor-version":[{"id":638,"href":"https:\/\/cristianku.com\/it\/wp-json\/wp\/v2\/posts\/234\/revisions\/638"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cristianku.com\/it\/wp-json\/wp\/v2\/media\/640"}],"wp:attachment":[{"href":"https:\/\/cristianku.com\/it\/wp-json\/wp\/v2\/media?parent=234"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cristianku.com\/it\/wp-json\/wp\/v2\/categories?post=234"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cristianku.com\/it\/wp-json\/wp\/v2\/tags?post=234"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}