potential performance improvements

[bstabler]

This issue is for keeping track of potential performance improvement ideas:

• reduce expression solving time and memory needs by better handling string data as pandas categorical data
• improve parallelization by taking advantage of updates to Python 3’s multiprocessing library
• continue to improve chunksize calculations for more optimized multiprocessing setups
• review ct-ramp and daysim performance ideas

Please add other ideas, thanks

[jpn--]

A configuration file switch that can disable trip-level processing for tours based on tour mode. So, you can shut off (skip) stop_frequency, trip_purpose, trip_destination, trip_scheduling, and trip_mode_choice for walk and bike tours if you don't care about those trips (e.g. inside a global feedback loop iteration, I don't care about walk or bike trips as they don't impact congestion).

Bonus points: the ability to easily flop the switch the other way, and re-start only the filtered tours (e.g. I decided I finished doing all my global feedback loops and I want those non-motorized trips back now)

[bstabler]

@stefancoe - add reading skim data from disk on-demand as opposed to reading every skim into RAM at the start as a way to trade runtime for RAM. @toliwaga implemented an undocumented version of this during the TVPB caching research and it runs slower but uses a lot less RAM. We may want to complete this feature for general use.

[bstabler]

Some more ideas from discussions with SANDAG:

• Move from strings to factors
• Exponentiate ahead of time TAP to TAP utilities, along with pre-computing access/egress costs
• Smarter binary search / picking of an alternative from a large choice set (such as for location choice)
• Make trip destination (i.e. intermediate stop location choice) aware of the tour mode so:
  o For bike, walk, transit to reduce the set of possible mazs ahead of time
  o For auto, to pre-compute TAZ to TAZ total utilities to avoid duplication of calculations
• Smarter chunking calculations to get more throughput Improve RAM/chunking settings #406
• Continued expression review/tidying up to reduce redundancy of calculations (i.e. optimization of written expressions)
• Buy a bigger / faster server and test ahead of time in the cloud what’s possible with respect to runtime reductions

[stefancoe]

Some good ideas here to increase pandas performance. The Pandas eval function looks interesting. Could it replace/substitute python eval in some cases?

[jpn--]

Could it replace/substitute python eval in some cases?

Not that we couldn't do it more, but we're already using pandas.eval in several places, for example:

• activitysim/activitysim/core/interaction_simulate.py

  Line 146 in bcdc7b6

  v = df.eval(expr)

• activitysim/activitysim/core/simulate.py

  Line 443 in bcdc7b6

  expression_values[i] = choosers.eval(expr)

[stefancoe]

Oh good to know-Thanks for pointing that out!

[bhargavasana]

Too broad. Some work completed on using categoricals #673. New, more specific issues could be opened in the future.