improve chunk_size setting by sub-model

[bstabler]

The full example is still running, but I've got the feeling that specifying a different chunk_size by sub-model would be a good idea in order to better performance tune the system.

[bstabler]

The full example completed this morning. It took 1027 minutes (or about 17 hours), which is quite a bit slower than last time (before the logsums revisions). The school and work location models should run a lot slower than before because they are now doing intelligent sampling and mode choice logsums. However, both cdap_simulate and non_mandatory_tour_frequency is now running significantly slower than before. All the other models are running either faster or about the same since @toliwaga made some other low level speedups. I'm guessing these models are running slower due to the revised chunk size. I tried a couple different chunk sizes (10m, 5m, and 1m) and was unable to set the chunk size greater than 1m since the workplace_location_sample model was running into memory problems (OverflowError: Python int too large to convert to C long). So, I think we should experiment with different chunk sizes by sub-model. Below are the runtimes for the full example run. Also, the output data pipeline is 30GB.

 Time to execute load skim_dict : 95.888 seconds (2.0 minutes)
 Time to execute run_model 'compute_accessibility' : 117.689 seconds (2.0 minutes)
 Time to execute run_model 'school_location_sample' : 1440.474 seconds (24.0 minutes)
 Time to execute run_model 'school_location_logsums' : 452.258 seconds (8.0 minutes)
 Time to execute run_model 'school_location_simulate' : 118.715 seconds (2.0 minutes)
 Time to execute run_model 'workplace_location_sample' : 13728.931 seconds (229.0 minutes)
 Time to execute run_model 'workplace_location_logsums' : 3748.511 seconds (62.0 minutes)
 Time to execute run_model 'workplace_location_simulate' : 326.275 seconds (5.0 minutes)
 Time to execute run_model 'auto_ownership_simulate' : 47.194 seconds (1.0 minutes)
 Time to execute run_model 'cdap_simulate' : 20663.336 seconds (344.0 minutes)
 Time to execute run_model 'mandatory_tour_frequency' : 711.669 seconds (12.0 minutes)
 Time to execute run_model 'mandatory_scheduling' : 3258.499 seconds (54.0 minutes)
 Time to execute run_model 'non_mandatory_tour_frequency' : 9823.686 seconds (164.0 minutes)
 Time to execute run_model 'destination_choice' : 901.387 seconds (15.0 minutes)
 Time to execute run_model 'non_mandatory_scheduling' : 4149.338 seconds (69.0 minutes)
 Time to execute run_model 'tour_mode_choice_simulate' : 642.547 seconds (11.0 minutes)
 Time to execute run_model 'create_simple_trips' : 49.801 seconds (1.0 minutes)
 Time to execute run_model 'trip_mode_choice_simulate' : 372.557 seconds (6.0 minutes)
 Time to execute all models : 61642.846 seconds (1027.0 minutes)

[toliwaga]

I am sure that both cdap_simulate and non_mandatory_tour_frequency will run as fast as before once the chunk sizing is adjusted.

Being able to set a model-specific chunk size might be handy in some cases, but it shouldn't be a replacement for getting chunk-sizing working correctly so that it performs well out-of-the-box.

That shouldn't be too hard. So far I have put no effort into tuning chunk sizing across model types. I just had to add something as the alternative set size was so large for logsums relative to other models.

[bstabler]

Over the weekend I ran a 2m chunk size run and it finished in 805 minutes. As expected, CDAP ran twice as fast as before, so let's plan to make chunk_size smarter by sub-model.

 Time to execute load skim_dict : 95.413 seconds (2.0 minutes)
 Time to execute run_model 'compute_accessibility' : 2.091 seconds (0.0 minutes)
 Time to execute run_model 'school_location_sample' : 1514.085 seconds (25.0 minutes)
 Time to execute run_model 'school_location_logsums' : 430.257 seconds (7.0 minutes)
 Time to execute run_model 'school_location_simulate' : 78.462 seconds (1.0 minutes)
 Time to execute run_model 'workplace_location_sample' : 13943.042 seconds (232.0 minutes)
 Time to execute run_model 'workplace_location_logsums' : 3549.856 seconds (59.0 minutes)
 Time to execute run_model 'workplace_location_simulate' : 276.208 seconds (5.0 minutes)
 Time to execute run_model 'auto_ownership_simulate' : 42.956 seconds (1.0 minutes)
 Time to execute run_model 'cdap_simulate' : 10359.343 seconds (173.0 minutes)
 Time to execute run_model 'mandatory_tour_frequency' : 664.236 seconds (11.0 minutes)
 Time to execute run_model 'mandatory_scheduling' : 3003.673 seconds (50.0 minutes)
 Time to execute run_model 'non_mandatory_tour_frequency' : 7781.711 seconds (130.0 minutes)
 Time to execute run_model 'destination_choice' : 752.222 seconds (13.0 minutes)
 Time to execute run_model 'non_mandatory_scheduling' : 3780.261 seconds (63.0 minutes)
 Time to execute run_model 'tour_mode_choice_simulate' : 633.605 seconds (11.0 minutes)
 Time to execute run_model 'create_simple_trips' : 49.972 seconds (1.0 minutes)
 Time to execute run_model 'trip_mode_choice_simulate' : 383.177 seconds (6.0 minutes)
 Time to execute all models : 48323.248 seconds (805.0 minutes)

[bstabler]

New full model runtimes

07/02/2018 14:54:58 -  run_model initialize) : 187.611 seconds (3.1 minutes)
07/02/2018 14:55:01 -  run_model compute_accessibility) : 2.495 seconds (0.0 minutes)
07/02/2018 15:21:04 -  run_model school_location_sample) : 1562.239 seconds (26.0 minutes)
07/02/2018 15:27:57 -  run_model school_location_logsums) : 412.602 seconds (6.9 minutes)
07/02/2018 15:29:32 -  run_model school_location_simulate) : 94.393 seconds (1.6 minutes)
07/02/2018 19:46:53 -  run_model workplace_location_sample) : 15440.354 seconds (257.3 minutes)
07/02/2018 22:51:13 -  run_model workplace_location_logsums) : 11059.663 seconds (184.3 minutes)
07/02/2018 22:56:24 -  run_model workplace_location_simulate) : 309.885 seconds (5.2 minutes)
07/02/2018 22:57:13 -  run_model auto_ownership_simulate) : 48.45 seconds (0.8 minutes)
07/02/2018 23:01:30 -  run_model cdap_simulate) : 255.914 seconds (4.3 minutes)
07/02/2018 23:03:57 -  run_model mandatory_tour_frequency) : 146.153 seconds (2.4 minutes)
08/02/2018 01:35:32 -  run_model mandatory_tour_scheduling) : 9094.887 seconds (151.6 minutes)
08/02/2018 03:11:56 -  run_model non_mandatory_tour_frequency) : 5783.455 seconds (96.4 minutes)
08/02/2018 03:24:09 -  run_model non_mandatory_tour_destination_choice) : 731.862 seconds (12.2 minutes)
08/02/2018 05:40:24 -  run_model non_mandatory_tour_scheduling) : 8174.29 seconds (136.2 minutes)
08/02/2018 05:41:44 -  run_model annotate_table.model_name=annotate_tours) : 79.921 seconds (1.3 minutes)
08/02/2018 06:02:07 -  run_model tour_mode_choice_simulate) : 1222.298 seconds (20.4 minutes)
08/02/2018 06:05:23 -  run_model atwork_subtour_frequency) : 194.4 seconds (3.2 minutes)
08/02/2018 06:18:08 -  run_model atwork_subtour_destination_sample) : 764.633 seconds (12.7 minutes)
08/02/2018 06:21:24 -  run_model atwork_subtour_destination_logsums) : 194.681 seconds (3.2 minutes)
08/02/2018 06:23:32 -  run_model atwork_subtour_destination_simulate) : 128.042 seconds (2.1 minutes)
08/02/2018 06:31:11 -  run_model atwork_subtour_scheduling) : 457.59 seconds (7.6 minutes)
08/02/2018 06:33:39 -  run_model atwork_subtour_mode_choice_simulate) : 147.753 seconds (2.5 minutes)
08/02/2018 06:35:02 -  run_model create_simple_trips) : 82.178 seconds (1.4 minutes)
08/02/2018 06:42:16 -  run_model trip_mode_choice_simulate) : 433.362 seconds (7.2 minutes)
08/02/2018 06:42:24 -  run_model write_data_dictionary) : 6.891 seconds (0.1 minutes)
08/02/2018 06:42:26 -  run_model write_tables) : 1.027 seconds (0.0 minutes)
08/02/2018 06:42:27 -  run (27 models) : 57036.334 seconds (950.6 minutes)