AttributeError: module 'collector' has no attribute 'YahooNormalizeUS1dExtend' while running update_data_to_bin

[rmallof]

🐛 Bug Description

After downloading all the US tickers I attempted to update the prices using 'python collector.py update_data_to_bin'

An observation made is it took almost the same amount of time than downloading all the data. Plus, by the end of the execution, the script crashed:

2022-07-13 06:22:32.848 | INFO | data_collector.base:_collector:197 - error symbol nums: 0
2022-07-13 06:22:32.848 | INFO | data_collector.base:_collector:198 - current get symbol nums: 12754
2022-07-13 06:22:32.848 | INFO | data_collector.base:collector_data:211 - 1 finish.
2022-07-13 06:22:32.848 | INFO | data_collector.base:collector_data:218 - total 12754, error: 0
Traceback (most recent call last):
File "collector.py", line 1203, in
fire.Fire(Run)
File "/opt/conda/envs/qlib/lib/python3.8/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/opt/conda/envs/qlib/lib/python3.8/site-packages/fire/core.py", line 466, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/opt/conda/envs/qlib/lib/python3.8/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "collector.py", line 1178, in update_data_to_bin
self.normalize_data_1d_extend(qlib_data_1d_dir)
File "collector.py", line 1058, in normalize_data_1d_extend
_class = getattr(self._cur_module, f"{self.normalize_class_name}Extend")
AttributeError: module 'collector' has no attribute 'YahooNormalizeUS1dExtend'

To Reproduce

Steps to reproduce the behavior:

1. Download all prices from US region
2. Run python collector.py update_data_to_bin --source_dir /us_source --normalize_dir /us_data_normal --qlib_data_1d_dir /us_data --region US --trading_date 2022-07-09 --end_date 2022-07-12

Expected Behavior

Prices are updated without crashing

Environment

Linux
x86_64
Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.10
#1 SMP Wed Mar 2 00:30:59 UTC 2022

Python version: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10) [GCC 10.3.0]

Qlib version: 0.8.6.99
numpy==1.23.1
pandas==1.4.3
scipy==1.8.1
requests==2.28.1
sacred==0.8.2
python-socketio==5.6.0
redis==4.3.4
python-redis-lock==3.7.0
schedule==1.1.0
cvxpy==1.2.1
hyperopt==0.1.2
fire==0.4.0
statsmodels==0.13.2
xlrd==2.0.1
plotly==5.5.0
matplotlib==3.5.1
tables==3.7.0
pyyaml==6.0
mlflow==1.27.0
tqdm==4.64.0
loguru==0.6.0
lightgbm==3.3.2
tornado==6.1
joblib==1.1.0
fire==0.4.0
ruamel.yaml==0.17.21

Additional Notes

[rmallof]

Maybe I'm missing something but the above command, even with --trading_date 2022-07-09 --end_date 2022-07-12, it downloaded data from 2000-12-28 to 2020-11-10

I'd appreciate some pointers on how to properly use the update_data_to_bin succesfully

Thank you!

[you-n-g]

Data updating is not supported for US data.
But I think not much customization is required.
You can follow this example and create a PR to make contributions to the community.

[rmallof]

Thanks for the reply, I'll check out the example above

Awesome project btw!