Merge improvements from SWITCH-WECC

[staadecker]

This issue will serve as a central point to discuss merging the issues from the SWITCH WECC repo.
As of now the plan is as follows.

Action plan

1. Move the SWITCH WECC latest version into a wecc branch in this repo.
2. Move all wecc specific files into a wecc package.
3. Create branches master-with-black-formatted-history and wecc-with-black-formatted-history which are the master and wecc branches rewritten but with their history modified such that every commit is formatted according to black. This reduces merge conflicts in the next step (see notes on how this was done below).
4. Merge master into wecc
5. Create merge requests one after another to include the changes from the wecc branch into the main branch as small features. These merge requests will have to be merged more or less in order. See chart below.

WECC features to include into master

• Graph generation (status: not started)
• Auto input data loading (status: not started)
• Tracking of other GHGs (status: not started)
• Variable scaling for numerical properties (status: not started)
• Addition of documentation (status: not started)
• Addition of switch drop tool (status: not started)
• Addition on some dual values in the output (status: not started)
• Improving README and docs (e.g. switch to .md format) (status: not started)
• Include wecc folder for others to see / use (status: not started)
• New policy modules (status: not started)
• Add switch compare (status: not started)

master features to include in WECC

• Specification of dimen in set definition

Conflicting changes

• Changes related to simultaneous upgrade to latest version of Pyomo
• Use of ordered=False in wecc but unique_list in master
• wecc specific examples e.g. ca_policies or stochastic example
• config.yaml approach to generating scenarios

Notes

Process for creating master-with-black-formatted-history and wecc-with-black-formatted-history

This is achieved with the following commands in Powershell.

1. Run git merge-base wecc master to find the commit where the two branches first diverged.

2. Checkout that commit on a new branch called common-wecc-parent.

3. Reformat the code with black --fast -t py310 --extend-exclude 'switch_wecc/wecc' ./switch_model and make a new commit.

4. Checkout master on switch to a new branch master-with-black-formatted-history.

5. Run git rebase --rebase-merges --exec "black --fast -t py310 --extend-exclude 'switch_wecc/wecc' ./switch_model; git commit -a --amend --allow-empty" common-wecc-parent
   a. The --rebase-merges ensures we keep the topography of the merges
   b. The --exec "black --fast -t py310 --extend-exclude 'switch_wecc/wecc' ./switch_model; git commit -a --amend --allow-empty" means that after every commit, we amend the commit such that the switch_model files are reformatted.

6. This will generate significant merge conflicts. For each one run
   a. git restore --theirs -s REBASE_HEAD . to update the local files to the commit that is being applied.
   b. black --fast -t py310 --extend-exclude 'switch_wecc/wecc' ./switch_model to reformat the files.
   c. git add . to mark any conflicts as resolved.
   d. git rebase --continue to continue with the rebase. To avoid the popup to edit the commit message see instructions here.
   To make these commands automatically run up to 100 times in a loop, in Powershell simply run
   1..10 | % {git restore --theirs -s REBASE_HEAD . ; black --fast -t py310 --extend-exclude 'switch_wecc/wecc' ./switch_model ; git add . ; git rebase --continue}

7. Check that there are no differences between master-with-black-formatted-history and master (you might need to run black on master)

8. Redo step 4 to 7 but for the wecc branch.

[staadecker]

Hello @PatyHidalgo @mfripp!

I spent most of my weekend exploring our options for joining the wecc branch with the master branch. There are several concerns that come to mind and make me think we might need to reconsider the approach. For context, I made both branches as similar as possible (i.e. reformatting both branches with black so that formatting differences don't cause merge conflicts, see above). Nonetheless, there are 87 files with merge conflicts that need to be resolved manually and 371 more files that would be added or changed due to a merge (some of which will require integration into the upstream switch system). My concerns are as follows:

1. REAM team bandwith. Including the latest changes from master into wecc will change the wecc code. Since I'm no longer an active user of switch-wecc, I would need support from your current students @PatyHidalgo to test these changes and make sure they don't break anything. Otherwise, it is very likely the changes will break the wecc version and your team won't migrate from the REAM repo to this repo. What is the bandwidth of your current team and what capabilities do they have in terms of being actively involved in fixing bugs and migrating their projects to new versions of SWITCH?
2. Matthias' bandwith. Given the large number of changes that we want to be integrated from wecc into master (see list above), we would need to keep a good pace on merge requests if we want to ever get this done. I see there are already several pending merge requests that are in need of attention. How busy are you at the moment @mfripp and are you able to dedicate more time weekly to get the changes merged?
3. New code and breaking behaviours for REAM team. A big concern is that fundamentally the wecc and master versions have approached things differently. There is significant new code required to merge the branches and the experience for your team @PatyHidalgo will need to change. For example, switch get_inputs would have to change to switch wecc get_inputs because we don't want all switch users to worry about get_inputs which is wecc specific. In this example, the change isn't complicated but it illustrates how some code will need to be re-written and I expect some other cases with be significantly more complicated. How would we approach resolving differences in approaches and how can we make sure your team @PatyHidalgo is constantly adapting to the new changes?
4. My bandwith. Finally, I am realizing that doing the full merge of both branches and making sure the changes don't break behaviour for users is something like a 2-week full time job (without considering back and forth discussion on which approaches to take). This might sound surprising however consider that since Aug 2019 (when the branches first diverged!) both the REAM and SWITCH repositories have gotten new changes that are often conflicting (e.g. both repositories upgraded to the latest version of Pyomo in slightly different ways). I would need to go through 3.5 years of changes one by one, fix any conflicts which often implies writing new code, and then test the change to make sure it doesn't break anything. Given I'm working a 50h/week job + trying to finish our paper @PatyHidalgo + have been asked to do paid work on a few other projects, I don't think I can commit to doing such a merge. Nor does it make much sense for me to be leading such a big project given that I'm no longer an active SWITCH user.

I see a few options:

1. Have both @PatyHidalgo team and @mfripp team do a multi-day "hackathon" ideally in-person to get the two version back together. Scheduling would be challenging but if the timing works I'm willing to join and support the team in merging the repos. This approach is what I think would most likely result in a success but also most intensive in terms of planning and resources.
2. Me working on merging a few key features from the wecc branch that @mfripp you really want in the main repository (with your support to get it tested and merged quickly before merge conflicts re-appear). The switch and switch-wecc would still be separate projects but at least the switch community could benefit from a few cool features.
3. Having someone from your team @PatyHidalgo who has more time and is a strong developer lead this effort. I could participate in e.g. a code review / mentoring role.
4. Look for funding to contract a developer (to be clear, not me) to do this merge (and potentially help with maintaining the repository). This could culminate in e.g. releasing a version 3.0 of SWITCH.
5. Let both versions of switch diverge and be independent projects. I don't think any of us want this option however I'm including it since we might conclude that given the magnitude of the resources required for this project, it is the most reasonable option.

Looking forward to figuring this out together! Working on open-source projects with limited resources is always a challenge but let's do the best we can. I would be very happy if the switch community can benefit from the switch-wecc features and if both teams can work on a unified project to make the benefits accessible to the entire community. I am sad that I can't lead this effort (I really would like to but I'm trying to make the smart decision and not overcommit myself as I usually do) but I will do what I can to help as much as possible!

[staadecker]

One more thing: I've thought of a good example to illustrate the challenge in resolving the differences between the switch-wecc and switch versions.

switch-wecc has scaled some of the core model's equations to improve the numerical properties and performance of its model. However, this change might be detrimental to another team depending on the exact numerical properties of their model. To merge this change their are 3 options:

1. Use the switch-wecc scaling but this might worsen the performance for other switch users.
2. Use the existing switch scaling, this will very likely worsen the performance for the switch-wecc model and REAM team.
3. Create a brand-new interface to allow the user to specify their own scaling (e.g. through an input file). This is the new code I'm talking about and is not as trivial and fixing some merge conflicts.
4. Maybe it's possible for the REAM team to rescale all their input parameters however this is an even bigger task.