当前位置:网站首页>Auto Sharding Policy will apply Data Sharding policy as it failed to apply file Sharding Policy
Auto Sharding Policy will apply Data Sharding policy as it failed to apply file Sharding Policy
2022-06-26 15:36:00 【There2belief】
Utilisertf.distribute.MirroredStrategy()Heure,Avertissement:
AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Did not find a shardable source, walked to a node which is not a dataset
Les codes sont les suivants::
strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
keras_model = build_model()
train_datagen = ImageDataGenerator()
training_img_generator = train_datagen.flow_from_directory(
input_path,
target_size=(image_size, image_size),
batch_size=batch_size,
class_mode="categorical",
)
train_dataset = tf.data.Dataset.from_generator(
lambda: training_img_generator,
output_types=(tf.float32, tf.float32),
output_shapes=([None, image_size, image_size, 3], [None, len(image_classes)])
)
# similar for validation_dataset = ...
keras_model.fit(
train_dataset,
steps_per_epoch=train_steps_per_epoch,
epochs=epoch_count,
validation_data=validation_dataset,
validation_steps=validation_steps_per_epoch,
)Now this seem to work, the model is trained as usual. However, during training I get the following warning message, when using a mirrored strategy:
AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Did not find a shardable source, walked to a node which is not a dataset
Tentative de résolution mais non valable
So I added the following lines between creating the data sets and calling fit():
options = tf.data.Options()
options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.DATA
train_dataset.with_options(options)
validation_dataset.with_options(options)However, I still get the same warning.
This leads me to these two questions:
- What do I need to do in order to get rid of this warning?
- Even more important: Why is TF not able to split the dataset with the default
AutoShardPolicy.FILEpolicy, since I am using thousands of images per class in the input folder?
边栏推荐
- Ansible自动化的运用
- 功能:crypto-js加密解密
- Shell script multi process concurrent writing method example (high level cultivation)
- English语法_形容词/副词3级 - 原级句型
- Summer camp is coming!!! Chongchongchong
- php文件上传00截断
- 【小程序实战系列】小程序框架 页面注册 生命周期 介绍
- [CEPH] Lock Notes of cephfs
- JS之手写 bind、apply、call
- Comparative analysis of restcloud ETL and kettle
猜你喜欢

【ceph】cephfs的锁 笔记
![[tcapulusdb knowledge base] tcapulusdb OMS business personnel permission introduction](/img/7b/8c4f1549054ee8c0184495d9e8e378.png)
[tcapulusdb knowledge base] tcapulusdb OMS business personnel permission introduction

Restcloud ETL resolves shell script parameterization

Particle filter PF - 3D CV target tracking with uniform motion (particle filter vs extended Kalman filter)

【微信小程序】事件绑定,你搞懂了吗?

如何配置使用新的单线激光雷达
Advanced operation of MySQL database basic SQL statement tutorial

Applet: uniapp solves vendor JS is too large

评价——模糊综合评价

RestCloud ETL抽取動態庫錶數據實踐
随机推荐
Applet: uniapp solves vendor JS is too large
音视频学习(二)——帧率、码流和分辨率
整理了一批脚本标准的函数模块(2021版)
Function: crypto JS encryption and decryption
Database - integrity constraints
HR export data Excel VBA
2022 Beijing Shijingshan District specializes in the application process for special new small and medium-sized enterprises, with a subsidy of 100000-200000 yuan
There are so many vulnerabilities in tcp/ip protocol?
Shell script multi process concurrent writing method example (high level cultivation)
selenium将元素保存为图片
Sikuli 基于图形识别的自动化测试技术
HW安全响应
About selenium common. exceptions. Webdriverexception: message: an unknown server side error solution (resolved)
RestCloud ETL抽取動態庫錶數據實踐
10 minutes to understand bim+gis fusion, common BIM data formats and characteristics
[C language practice - printing hollow upper triangle and its deformation]
Comparative analysis of restcloud ETL and kettle
IDEA本地代理后,无法下载插件
【TcaplusDB知识库】TcaplusDB单据受理-创建业务介绍
【leetcode】48.旋转图像