You have been asked to develop an input pipeline for an ML training model that processes images from disparate sources at a low latency.
You discover that your input data does not fit in memory.
How should you create a dataset following Google-recommended best practices?
Click on the arrows to vote for the correct answer
A. B. C. D.B.
https://www.tensorflow.org/api_docs/python/tf/data/DatasetWhen dealing with large datasets that do not fit in memory, it is important to use an input pipeline that can efficiently process the data at a low latency. Google recommends using the TensorFlow's tf.data API for building efficient and scalable input pipelines.
In the given scenario, the input data consists of images from disparate sources that do not fit in memory. Therefore, we need to create a dataset that can efficiently process the data without requiring it to be loaded into memory at once. The recommended approach for this scenario is to convert the images into TFRecords, store the images in Cloud Storage, and then use the tf.data API to read the images for training.
Option D is the correct answer, and the following explanation justifies this choice:
Option A - Create a tf.data.Dataset.prefetch transformation: The tf.data.Dataset.prefetch() transformation allows the dataset to asynchronously fetch batches of data in the background while the model is training on the current batch. While prefetching can improve the performance of the input pipeline, it does not address the issue of large datasets that do not fit in memory.
Option B - Convert the images to tf.Tensor objects, and then run Dataset.from_tensor_slices(): The tf.Tensor objects represent a tensor of fixed shape and data type. Converting the images to tensors may not be feasible, especially if the images are of different sizes or aspect ratios. Also, using Dataset.from_tensor_slices() would require loading all the data into memory, which is not possible in this scenario.
Option C - Convert the images to tf.Tensor objects, and then run tf.data.Dataset.from_tensors(): Similar to Option B, converting the images to tensors and using tf.data.Dataset.from_tensors() would require loading all the data into memory, which is not possible in this scenario.
Option D - Convert the images into TFRecords, store the images in Cloud Storage, and then use the tf.data API to read the images for training: This option is the recommended approach for processing large datasets that do not fit in memory. By converting the images to TFRecords, we can store them in Cloud Storage in a compressed format, which reduces the amount of storage required. The tf.data API can then be used to read the images in parallel, batch them, and preprocess them on the fly. This approach allows us to process large datasets efficiently and at a low latency.
In summary, Option D is the correct answer as it follows the Google-recommended best practices for creating an input pipeline that can process large datasets of images from disparate sources that do not fit in memory.