This is the implementation: import concurrent.futures I would say that I see a significant improvement on the reading time, however this also depend as I mentioned before on the network. I have tried(having a lot of doubts how works) the concurrent.futures module. This is the current way of importing files: import pandas as pdĭf1 = pd.read_csv(r'\.\path_to_file', sep='\t', header=0,ĭf2 = pd.read_excel(r'\.\path_to_file', Unfortunately other formats like HDF5 are not an option at least by now. I am trying to speed up this process, however one of the limitation that I have is that I cannot change the format, so it has to be xlsx format or csv.
There is a high dependency on the read time because of the network, it can go from 150 sec. I have an script on my computer that reads files(.xlsx.csv) from a network shared drive.
0 kommentar(er)
